search

hashicorp / terraform

Terraform enables you to safely and predictably create, change, and improve infrastructure. It is a source-available tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.
https://www.terraform.io/
Other
42.45k stars 9.51k forks source link

Terraform S3 remote state tries to access env:/ key (reversion?) #22492

Open markchalloner opened 5 years ago

markchalloner commented 5 years ago

Terraform Version

Terraform v0.12.6
+ provider.archive v1.2.2
+ provider.aws v2.14.0

Terraform Configuration Files

terraform {
  required_version = ">= 0.12"
  backend "s3" {
    region         = "eu-west-1"
    encrypt        = true
    dynamodb_table = "{redacted}"
    bucket         = "{redacted}"
    key            = "{redacted}"
  }
}

Debug Output

2019/08/16 17:31:43 [INFO] Terraform version: 0.12.6                                                                                                          
2019/08/16 17:31:43 [INFO] Go runtime version: go1.12.4                                                                                                       
2019/08/16 17:31:43 [INFO] CLI args: []string{"{redacted}/terraform", "apply"}                                       
2019/08/16 17:31:43 [DEBUG] Attempting to open CLI config file: {redacted}/.terraformrc                                                                       
2019/08/16 17:31:43 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.                                                                                
2019/08/16 17:31:43 [INFO] CLI command args: []string{"apply"}                                                                                                
2019/08/16 17:31:43 [TRACE] Meta.Backend: built configuration for "s3" backend with hash value 1579759270                                                     
2019/08/16 17:31:43 [TRACE] Preserving existing state lineage "{redacted}"                                                          
2019/08/16 17:31:43 [TRACE] Preserving existing state lineage "{redacted}"                                                          
2019/08/16 17:31:43 [TRACE] Meta.Backend: working directory was previously initialized for "s3" backend                                                       
2019/08/16 17:31:43 [TRACE] Meta.Backend: using already-initialized, unchanged "s3" backend configuration                                                     
2019/08/16 17:31:43 [INFO] Setting AWS metadata API timeout to 100ms                                                                                          
2019/08/16 17:31:44 [INFO] Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id                                        
2019/08/16 17:31:44 [INFO] AWS Auth provider used: "EnvProvider"                                                                                              
2019/08/16 17:31:44 [DEBUG] Trying to get account information via sts:GetCallerIdentity                                                                       
2019/08/16 17:31:44 [DEBUG] [aws-sdk-go] DEBUG: Request sts/GetCallerIdentity Details:                                                                        
---[ REQUEST POST-SIGN ]-----------------------------                                                                                                         
POST / HTTP/1.1                                                                                                                                               
Host: sts.amazonaws.com                                                                                                                                       
User-Agent: aws-sdk-go/1.21.7 (go1.12.4; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.12.6                                                                 
Content-Length: 43                                                                                                                                            
Authorization: {redacted}
Content-Type: application/x-www-form-urlencoded; charset=utf-8
X-Amz-Date: 20190816T163144Z
Accept-Encoding: gzip

Action=GetCallerIdentity&Version=2011-06-15
-----------------------------------------------------
2019/08/16 17:31:45 [DEBUG] [aws-sdk-go] DEBUG: Response sts/GetCallerIdentity Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Connection: close
Content-Length: 426
Content-Type: text/xml
Date: Fri, 16 Aug 2019 16:31:44 GMT
X-Amzn-Requestid: {redacted}

-----------------------------------------------------
2019/08/16 17:31:45 [DEBUG] [aws-sdk-go] <GetCallerIdentityResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">                                        
  <GetCallerIdentityResult>                                                                                                                                   
    <Arn>{redacted}</Arn>
    <UserId>{redacted}</UserId>
    <Account>{redacted}</Account>
  </GetCallerIdentityResult>
  <ResponseMetadata>
    <RequestId>{redacted}</RequestId>
  </ResponseMetadata>
</GetCallerIdentityResponse>
2019/08/16 17:31:45 [TRACE] Meta.Backend: instantiated backend of type *s3.Backend
2019/08/16 17:31:45 [DEBUG] checking for provider in "."
2019/08/16 17:31:45 [DEBUG] checking for provider in "{redacted}"
2019/08/16 17:31:45 [DEBUG] checking for provider in ".terraform/plugins/linux_amd64"
2019/08/16 17:31:45 [DEBUG] found provider "terraform-provider-archive_v1.2.2_x4"
2019/08/16 17:31:45 [DEBUG] found provider "terraform-provider-aws_v2.14.0_x4"
2019/08/16 17:31:45 [DEBUG] found valid plugin: "aws", "2.14.0", "{redacted}/.terraform/plugins/linux_amd64/terraform-provider-aws_v2.14.0_x4"
2019/08/16 17:31:45 [DEBUG] found valid plugin: "archive", "1.2.2", "{redacted}/.terraform/plugins/linux_amd64/terraform-provider-archive_v1.2.2_x4"
2019/08/16 17:31:45 [DEBUG] checking for provisioner in "."
2019/08/16 17:31:45 [DEBUG] checking for provisioner in "{redacted}"
2019/08/16 17:31:45 [DEBUG] checking for provisioner in ".terraform/plugins/linux_amd64"
2019/08/16 17:31:45 [TRACE] Meta.Backend: backend *s3.Backend does not support operations, so wrapping it in a local backend
2019/08/16 17:31:45 [INFO] backend/local: starting Apply operation
2019/08/16 17:31:45 [TRACE] backend/local: requesting state manager for workspace "default"
2019/08/16 17:31:45 [DEBUG] [aws-sdk-go] DEBUG: Request s3/ListObjects Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /?prefix=env%3A%2F HTTP/1.1
Host: {redacted}.s3.eu-west-1.amazonaws.com
User-Agent: aws-sdk-go/1.21.7 (go1.12.4; linux; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.12.6
Authorization: {redacted}
X-Amz-Content-Sha256: {redacted}
X-Amz-Date: 20190816T163145Z
Accept-Encoding: gzip

-----------------------------------------------------
2019/08/16 17:31:45 [DEBUG] [aws-sdk-go] DEBUG: Response s3/ListObjects Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 403 Forbidden
Connection: close
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Fri, 16 Aug 2019 16:31:44 GMT
Server: AmazonS3
X-Amz-Bucket-Region: eu-west-1
X-Amz-Id-2: {redacted}
X-Amz-Request-Id: {redacted}

-----------------------------------------------------
2019/08/16 17:31:45 [DEBUG] [aws-sdk-go] <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>{redacted}</RequestId><HostId>{redacted}</HostId></Error>
2019/08/16 17:31:45 [DEBUG] [aws-sdk-go] DEBUG: Validate Response s3/ListObjects failed, attempt 0/5, error AccessDenied: Access Denied
        status code: 403, request id: {redacted}, host id: {redacted}

Error: Error loading state: AccessDenied: Access Denied
        status code: 403, request id: {redacted}, host id: {redacted}

Crash Output

Expected Behavior

No attempt to access env:/ key

Actual Behavior

Attempted to access env:/ key:

GET /?prefix=env%3A%2F HTTP/1.1

Steps to Reproduce

  1. terraform init
  2. terraform apply

Additional Context

References

john-housser commented 4 years ago

I just stumbled into the same issue. It looks like "env:" is the default workspace_key_prefix that gets prepended onto your workspace name for non-default workspaces, as a prefix to the key.

In my case, it seemed easiest for me to use project-specific workspace_key_prefixes rather than project-specific keys, and I'm just using "state" for the key for all my projects that share the same bucket, but ensuring they have unique workspace_key_prefixes so that I can scope S3 permissions to just that S3 path.

cmawhorter commented 3 years ago

If you hit this issue and nothing else is working, make sure the actual object is owned/accessible to the profile you're using.

In my case the state was pushed to the bucket under another aws account for some reason and this was preventing terraform from getting it using the profile I provided. The errors and log entries made it seem like it was something else. Once I reassumed ownership of the state file things worked again.