search

hashicorp / terraform

Terraform enables you to safely and predictably create, change, and improve infrastructure. It is a source-available tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.
https://www.terraform.io/
Other
42.73k stars 9.56k forks source link

Corrupted state in 1.6.1 with S3 backend on nonAWS #34054

Closed legioner0 closed 1 year ago

legioner0 commented 1 year ago

Terraform Version

Terraform v1.6.1
on linux_amd64

Terraform Configuration Files

terraform {
  required_version = ">= 1.6.1"

  backend "s3" {
    bucket                      = "<bucket>"
    key                         = "<key>"
    region                      = "<region>"
    skip_region_validation      = true
    skip_credentials_validation = true
    skip_requesting_account_id  = true
    skip_metadata_api_check     = true
    use_path_style              = true
    profile                     = "<profile>"
    endpoints                   = {
      s3 = "https://<custom s3 service endpoint>"
    }
  }
}

Debug Output

export TF_LOG=trace ; terraform apply (with empty state)

...
2023-10-11T12:01:53.031+0700 [DEBUG] states/remote: state read serial is: 0; serial is: 0
2023-10-11T12:01:53.031+0700 [DEBUG] states/remote: state read lineage is: ; lineage is:
2023-10-11T12:01:53.031+0700 [INFO]  backend-s3: Downloading remote state: tf_backend.operation=Get tf_backend.req_id=7d1254b6-ac8a-1af3-d9a0-e9029b161c0d tf_backend.s3.bucket=<bucket> tf_backend.s3.path=<key>
2023-10-11T12:01:53.031+0700 [DEBUG] backend-s3: HTTP Request Sent: aws.custom_endpoint_source=true aws.operation=HeadObject aws.region=<region> aws.sdk=aws-sdk-go-v2 aws.service=S3 tf_backend.operation=Get tf_backend.req_id=7d1254b6-ac8a-1af3-d9a0-e9029b161c0d tf_backend.s3.bucket=<bucket> tf_backend.s3.path=<key> http.request.header.x_amz_content_sha256=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 http.request.header.x_amz_date=20231011T050153Z http.request.body="" http.request.header.amz_sdk_request="attempt=1; max=5" http.request.header.amz_sdk_invocation_id=7241b840-ff47-410e-8c50-7d11e40c46e3 http.method=HEAD http.url=https://<endpoint>/<bucket>/<key> net.peer.name=<endpoint> http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.6.1 (+https://www.terraform.io) aws-sdk-go-v2/1.21.0 os/linux lang/go#1.21.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.38.5" http.request.header.authorization="AWS4-HMAC-SHA256 Credential=<hidden>, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=*****" http.request.header.accept_encoding=identity
2023-10-11T12:01:53.160+0700 [DEBUG] backend-s3: HTTP Response Received: aws.custom_endpoint_source=true aws.operation=HeadObject aws.region=<region> aws.sdk=aws-sdk-go-v2 aws.service=S3 tf_backend.operation=Get tf_backend.req_id=7d1254b6-ac8a-1af3-d9a0-e9029b161c0d tf_backend.s3.bucket=<bucket> tf_backend.s3.path=<key> http.status_code=404 http.response_content_length=231 http.response.header.date="Wed, 11 Oct 2023 05:01:53 GMT" http.response.body="" http.duration=129 http.response.header.x_container_storage_policy_index=1 http.response.header.x_container_storage_policy_name=cold http.response.header.content_type="text/xml; charset=utf-8" http.response.header.x_amz_request_id=802a41fb-fae9-4fcc-93bb-e9093bae37ea
2023-10-11T12:01:53.161+0700 [DEBUG] backend-s3: request failed with unretryable error https response error StatusCode: 404, RequestID: 802a41fb-fae9-4fcc-93bb-e9093bae37ea, HostID: , NotFound: : tf_backend.operation=Get tf_backend.req_id=7d1254b6-ac8a-1af3-d9a0-e9029b161c0d tf_backend.s3.bucket=<bucket> tf_backend.s3.path=<key>
2023-10-11T12:01:53.161+0700 [DEBUG] states/remote: after refresh, state read serial is: 0; serial is: 0
2023-10-11T12:01:53.161+0700 [DEBUG] states/remote: after refresh, state read lineage is: ; lineage is:
2023-10-11T12:01:53.161+0700 [INFO]  backend-s3: Uploading remote state: tf_backend.operation=Put tf_backend.req_id=c51d29ab-0535-c91a-5f4a-67ca29000578 tf_backend.s3.bucket=<bucket> tf_backend.s3.path=<key>
2023-10-11T12:01:53.163+0700 [DEBUG] backend-s3: HTTP Request Sent: aws.custom_endpoint_source=true aws.operation=PutObject aws.region=<region> aws.sdk=aws-sdk-go-v2 aws.service=S3 tf_backend.operation=Put tf_backend.req_id=c51d29ab-0535-c91a-5f4a-67ca29000578 tf_backend.s3.bucket=<bucket> tf_backend.s3.path=<key> http.url=https://<endpoint>/<bucket>/<key>?x-id=PutObject http.request.header.authorization="AWS4-HMAC-SHA256 Credential=<hidden>, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-encoding;content-length;content-type;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-sdk-checksum-algorithm;x-amz-trailer, Signature=*****" http.request.header.x_amz_sdk_checksum_algorithm=SHA256 http.request.header.accept_encoding=identity http.request.header.content_encoding=aws-chunked http.request.header.x_amz_date=20231011T050153Z http.method=PUT http.request.header.content_type=application/json http.request.header.amz_sdk_invocation_id=9d8de940-132f-484d-9b43-ee48fd718a26 net.peer.name=<endpoint>
  http.request.body=
  | b4
  | {
  |   "version": 4,
  |   "terraform_version": "1.6.1",
  |   "serial": 1,
  |   "lineage": "dbe9de12-d1b8-5a7f-83e9-d619219cd6f8",
  |   "outputs": {},
  |   "resources": [],
  |   "check_results": null
  | }
  |
  | 0
  | x-amz-checksum-sha256:xCIxJnT7YQ9mVY7ZDMAB2gJ8Hy6XkHCCsljQt/shx94=
  |
   http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.6.1 (+https://www.terraform.io) aws-sdk-go-v2/1.21.0 os/linux lang/go#1.21.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.38.5 ft/s3-transfer" http.request.header.x_amz_decoded_content_length=180 http.request.header.x_amz_trailer=x-amz-checksum-sha256 http.request.header.amz_sdk_request="attempt=1; max=5" http.request.header.x_amz_content_sha256=STREAMING-UNSIGNED-PAYLOAD-TRAILER http.request_content_length=259
2023-10-11T12:01:53.424+0700 [DEBUG] backend-s3: HTTP Response Received: aws.custom_endpoint_source=true aws.operation=PutObject aws.region=<region> aws.sdk=aws-sdk-go-v2 aws.service=S3 tf_backend.operation=Put tf_backend.req_id=c51d29ab-0535-c91a-5f4a-67ca29000578 tf_backend.s3.bucket=<bucket> tf_backend.s3.path=<key> http.response.header.x_container_storage_policy_name=cold http.response.header.date="Wed, 11 Oct 2023 05:01:53 GMT" http.response.header.etag="\"759e4d837097fe86d79a5dfaa687e311\"" http.response.header.x_amz_request_id=a9b0fcaa-d137-4269-82f6-e190b1e39f2e http.response.header.x_container_storage_policy_index=1 http.response.body="" http.duration=261 http.status_code=200
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

export TF_LOG=trace ; terraform init (second call after apply)

...
2023-10-11T12:11:48.704+0700 [DEBUG] backend-s3: HTTP Request Sent: aws.custom_endpoint_source=true aws.operation=ListObjectsV2 aws.region=<region> aws.sdk=aws-sdk-go-v2 aws.service=S3 tf_backend.operation=Workspaces tf_backend.req_id=2a537b06-9448-b7a4-9fb3-8ed255113e4c tf_backend.s3.bucket=<bucket> tf_backend.workspace-prefix=env:/ http.request.header.amz_sdk_invocation_id=1b67598b-c591-4066-a146-dea6822cc610 http.request.header.x_amz_content_sha256=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 http.request.header.amz_sdk_request="attempt=1; max=5" http.request.header.x_amz_date=20231011T051148Z http.method=GET http.url="https://<endpoint>/<bucket>?list-type=2&max-keys=1000&prefix=env%3A%2F" net.peer.name=<endpoint> http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.6.1 (+https://www.terraform.io) aws-sdk-go-v2/1.21.0 os/linux lang/go#1.21.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.38.5" http.request.header.authorization="AWS4-HMAC-SHA256 Credential=<hidden>, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=*****" http.request.header.accept_encoding=identity http.request.body=""
2023-10-11T12:11:48.843+0700 [DEBUG] backend-s3: HTTP Response Received: aws.custom_endpoint_source=true aws.operation=ListObjectsV2 aws.region=<region> aws.sdk=aws-sdk-go-v2 aws.service=S3 tf_backend.operation=Workspaces tf_backend.req_id=2a537b06-9448-b7a4-9fb3-8ed255113e4c tf_backend.s3.bucket=<bucket> tf_backend.workspace-prefix=env:/ http.response_content_length=265 http.duration=138 http.response.header.content_type=application/xml http.response.header.x_amz_request_id=f1a644d5-b289-4837-b800-6c4d89daa817 http.response.header.x_container_storage_policy_index=1 http.response.header.x_container_storage_policy_name=cold http.response.header.date="Wed, 11 Oct 2023 05:11:48 GMT"
  http.response.body=
  | <?xml version="1.0" encoding="UTF-8"?>
  | <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Delimiter></Delimiter><Prefix>env:/</Prefix><Name><bucket></Name><KeyCount>0</KeyCount><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated></ListBucketResult>
   http.status_code=200
2023-10-11T12:11:48.843+0700 [INFO]  backend-s3: Downloading remote state: tf_backend.operation=Get tf_backend.req_id=a6d9a127-cd80-93e5-08e4-20a0b55fe9fc tf_backend.s3.bucket=<bucket> tf_backend.s3.path=<key>
2023-10-11T12:11:48.844+0700 [DEBUG] backend-s3: HTTP Request Sent: aws.custom_endpoint_source=true aws.operation=HeadObject aws.region=<region> aws.sdk=aws-sdk-go-v2 aws.service=S3 tf_backend.operation=Get tf_backend.req_id=a6d9a127-cd80-93e5-08e4-20a0b55fe9fc tf_backend.s3.bucket=<bucket> tf_backend.s3.path=<key> net.peer.name=<endpoint> http.request.header.amz_sdk_invocation_id=a69c93ec-5ace-49ae-a3fc-6a5def223521 http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.6.1 (+https://www.terraform.io) aws-sdk-go-v2/1.21.0 os/linux lang/go#1.21.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.38.5" http.request.header.authorization="AWS4-HMAC-SHA256 Credential=<hidden>, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=*****" http.request.header.x_amz_content_sha256=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 http.request.header.accept_encoding=identity http.request.header.amz_sdk_request="attempt=1; max=5" http.request.header.x_amz_date=20231011T051148Z http.method=HEAD http.url=https://<endpoint>/<bucket>/<key> http.request.body=""
2023-10-11T12:11:48.974+0700 [DEBUG] backend-s3: HTTP Response Received: aws.custom_endpoint_source=true aws.operation=HeadObject aws.region=<region> aws.sdk=aws-sdk-go-v2 aws.service=S3 tf_backend.operation=Get tf_backend.req_id=a6d9a127-cd80-93e5-08e4-20a0b55fe9fc tf_backend.s3.bucket=<bucket> tf_backend.s3.path=<key> http.status_code=200 http.response_content_length=259 http.response.header.etag="\"759e4d837097fe86d79a5dfaa687e311\"" http.response.header.last_modified="Wed, 11 Oct 2023 05:01:53 GMT" http.response.header.date="Wed, 11 Oct 2023 05:11:48 GMT" http.response.header.content_type=application/json http.duration=129 http.response.header.x_container_storage_policy_index=1 http.response.header.x_amz_request_id=addfe3f3-47f2-4194-8ccc-5e9e0495668c http.response.header.x_container_storage_policy_name=cold http.response.body="" http.response.header.accept_ranges=bytes
2023-10-11T12:11:48.975+0700 [DEBUG] backend-s3: HTTP Request Sent: aws.custom_endpoint_source=true aws.operation=GetObject aws.region=<region> aws.sdk=aws-sdk-go-v2 aws.service=S3 tf_backend.operation=Get tf_backend.req_id=a6d9a127-cd80-93e5-08e4-20a0b55fe9fc tf_backend.s3.bucket=<bucket> tf_backend.s3.path=<key> http.method=GET http.url=https://<endpoint>/<bucket>/<key>?x-id=GetObject http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.6.1 (+https://www.terraform.io) aws-sdk-go-v2/1.21.0 os/linux lang/go#1.21.1 md/GOOS#linux md/GOARCH#amd64 api/s3#1.38.5 ft/s3-transfer" http.request.header.authorization="AWS4-HMAC-SHA256 Credential=<hidden>, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;range;x-amz-content-sha256;x-amz-date, Signature=*****" http.request.header.accept_encoding=identity http.request.header.x_amz_date=20231011T051148Z http.request.header.amz_sdk_invocation_id=4693cdf2-fad0-468e-b158-5db1346e587f net.peer.name=<endpoint> http.request.header.amz_sdk_request="attempt=1; max=5" http.request.header.range=bytes=0-5242879 http.request.header.x_amz_content_sha256=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 http.request.body=""
2023-10-11T12:11:49.124+0700 [DEBUG] backend-s3: HTTP Response Received: aws.custom_endpoint_source=true aws.operation=GetObject aws.region=<region> aws.sdk=aws-sdk-go-v2 aws.service=S3 tf_backend.operation=Get tf_backend.req_id=a6d9a127-cd80-93e5-08e4-20a0b55fe9fc tf_backend.s3.bucket=<bucket> tf_backend.s3.path=<key> http.response.header.content_range="bytes 0-258/259" http.response.header.accept_ranges=bytes http.response.header.x_amz_request_id=b816eeb4-b270-4fe8-8c53-982653f678bc http.duration=148 http.response.header.x_container_storage_policy_index=1 http.response.header.date="Wed, 11 Oct 2023 05:11:49 GMT" http.response.header.content_type=application/json
  http.response.body=
  | b4
  | {
  |   "version": 4,
  |   "terraform_version": "1.6.1",
  |   "serial": 1,
  |   "lineage": "dbe9de12-d1b8-5a7f-83e9-d619219cd6f8",
  |   "outputs": {},
  |   "resources": [],
  |   "check_results": null
  | }
  | 
  | 0
  | x-amz-checksum-sha256:xCIxJnT7YQ9mVY7ZDMAB2gJ8Hy6XkHCCsljQt/shx94=
  | 
   http.status_code=206 http.response.header.x_container_storage_policy_name=cold http.response.header.last_modified="Wed, 11 Oct 2023 05:01:53 GMT" http.response_content_length=259 http.response.header.etag="\"759e4d837097fe86d79a5dfaa687e311\""
Error refreshing state: 2 problems:

- Unsupported state file format: The state file could not be parsed as JSON: syntax error at byte offset 1.
- Unsupported state file format: The state file does not have a "version" attribute, which is required to identify the format version.

Expected Behavior

correct state file on s3

Actual Behavior

terraform apply save corrupted state file on S3 backend. As far as I can see: data before json - size of json, data after json - header with sha256 checksum.

Steps to Reproduce

  1. terraform init
  2. terraform apply
  3. terraform init

Additional Context

This environment uses S3-compatible service provider.

References

No response

gdavison commented 1 year ago

@legioner0, can you confirm if the object in Ceph contains just the JSON body or the full payload shown below?

b4
{
  "version": 4,
  "terraform_version": "1.6.1",
  "serial": 1,
  "lineage": "dbe9de12-d1b8-5a7f-83e9-d619219cd6f8",
  "outputs": {},
  "resources": [],
  "check_results": null
}

0
x-amz-checksum-sha256:xCIxJnT7YQ9mVY7ZDMAB2gJ8Hy6XkHCCsljQt/shx94=

If it contains the full payload, it looks like Ceph is not correctly handling aws-chunked encoding, which is required for chunked transfers with AWS Signature v4. See https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html for more information.

legioner0 commented 1 year ago

@gdavison, yes state file on s3-compatible bucket contains full payload with size and x-amz-checksum-sha256 line. I suppose, that parameter skip_s3_checksum (#34127) will fix problem.

legioner0 commented 1 year ago

Confirmed, skip_s3_checksum fix the issue with S3-compatible service.

terraform {
  required_version = ">= 1.6.3"
  backend "s3" {
    bucket                      = "<bucket>"
    key                         = "<key>"
    region                      = "<region>"
    skip_region_validation      = true
    skip_credentials_validation = true
    skip_requesting_account_id  = true
    skip_metadata_api_check     = true
    skip_s3_checksum            = true
    use_path_style              = true
    profile                     = "<profile>"
    endpoints                   = {
      s3 = "https://<custom s3 service endpoint>"
    }
  }
}
crw commented 1 year ago

@legioner0 is it OK to close this issue?

legioner0 commented 1 year ago

yes, thanks

github-actions[bot] commented 11 months ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.