Terraform 1.6.x does not find *.tfstate file in s3 which was created using older versions of terraform

[jurna]

We have a project with submodules support. Currently we have terragrunt.hcl file that looks like this:

remote_state {
  backend  = "s3"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
  config   = {
    bucket         = "terraform-bucket"
    key            = "${path_relative_to_include()}/terraform.tfstate"
    region         = "eu-central-1"
    encrypt        = true
    dynamodb_table = "terraform-state-lock"
  }
}

This configuration created terraform.tfstate file in s3 using terraform version 1.5.x

Now when tyring to do anything with terraform version 1.6.x, terraform does not find tfstate file. It searchies for './terraform.tfstate' file instead of 'terraform.tfstate' and fails.

Here is log from terraform 1.5.x (credentials removed) (result - 200 OK):

-----------------------------------------------------
2023-11-15T12:11:55.779+0200 [DEBUG] [aws-sdk-go] {}
2023-11-15T12:11:55.780+0200 [DEBUG] [aws-sdk-go] DEBUG: Request s3/GetObject Details:
---[ REQUEST POST-SIGN ]-----------------------------
GET /terraform.tfstate HTTP/1.1
Host: terraform-bucket.s3.eu-central-1.amazonaws.com
User-Agent: APN/1.0 HashiCorp/1.0 Terraform/1.5.7 aws-sdk-go/1.44.122 (go1.20.7; linux; amd64)
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20231115T101155Z
Accept-Encoding: gzip

-----------------------------------------------------
2023-11-15T12:11:55.991+0200 [DEBUG] [aws-sdk-go] DEBUG: Response s3/GetObject Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Connection: close
Content-Length: 384760
Accept-Ranges: bytes
Content-Type: application/json
Date: Wed, 15 Nov 2023 10:11:56 GMT
Etag: "f357341b9645d1643cb8ee79022cd215"
Last-Modified: Fri, 27 Oct 2023 08:46:51 GMT
Server: AmazonS3
X-Amz-Id-2: Y9LAIgVKl5bjua2wExd1quq5KXdnDq5wCd8eE3GbFJlqo3bNOkVvqjoZITgeZkp9JuMHn8wRw0E=
X-Amz-Request-Id: BQPC6ES63540GTHN
X-Amz-Server-Side-Encryption: AES256
X-Amz-Version-Id: nhKueNmyZNat3tqAlmwAeGFTdcYYwWPN

Here is the log from terraform 1.6.x (404 error):

2023-11-15T12:15:07.874+0200 [INFO]  backend-s3: Downloading remote state: tf_backend.operation=Get tf_backend.req_id=c6b7a8aa-e870-7d70-8964-52935f98b12d tf_backend.s3.bucket=terraform-bucket tf_backend.s3.path=./terraform.tfstate
2023-11-15T12:15:07.875+0200 [DEBUG] backend-s3: HTTP Request Sent: aws.operation=HeadObject aws.region=eu-central-1 aws.sdk=aws-sdk-go-v2 aws.service=S3 tf_backend.operation=Get tf_backend.req_id=c6b7a8aa-e870-7d70-8964-52935f98b12d tf_backend.s3.bucket=terraform-bucket tf_backend.s3.path=./terraform.tfstate http.request.header.x_amz_date=20231115T101507Z http.method=HEAD net.peer.name=terraform-bucket.s3.eu-central-1.amazonaws.com http.user_agent="APN/1.0 HashiCorp/1.0 Terraform/1.6.3 (+https://www.terraform.io) aws-sdk-go-v2/1.21.0 os/linux lang/go#1.21.3 md/GOOS#linux md/GOARCH#amd64 api/s3#1.38.5" http.request.header.authorization="AWS4-HMAC-SHA256 Credential=AKIA************AKO7/20231115/eu-central-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;host;x-amz-content-sha256;x-amz-date, Signature=*****" http.request.header.x_amz_content_sha256=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 http.url=https://terraform-bucket.s3.eu-central-1.amazonaws.com/./terraform.tfstate http.request.header.amz_sdk_invocation_id=813d4010-7c72-4ccf-a30e-59331be39944 http.request.header.accept_encoding=identity http.request.header.amz_sdk_request="attempt=1; max=5" http.request.body=""
2023-11-15T12:15:07.921+0200 [DEBUG] backend-s3: HTTP Response Received: aws.operation=HeadObject aws.region=eu-central-1 aws.sdk=aws-sdk-go-v2 aws.service=S3 tf_backend.operation=Get tf_backend.req_id=c6b7a8aa-e870-7d70-8964-52935f98b12d tf_backend.s3.bucket=terraform-bucket tf_backend.s3.path=./terraform.tfstate http.response.header.date="Wed, 15 Nov 2023 10:15:07 GMT" http.response.header.server=AmazonS3 http.response.body="" http.duration=46 http.status_code=404 http.response.header.x_amz_request_id=NX3QC1MC0N9N5TW6 http.response.header.x_amz_id_2="yUmgCqK3zpSB0bReJ5bxxCy0B88ZhARUw5Omn5VsbCUzGSUbADoqnOBurOZZ+GL0a6eDbY7gl4Q=" http.response.header.content_type=application/xml

[zhaochunqi]

same here, changing to 1.5.x solved the problem.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for raising this issue.

[zhaochunqi]

Has this problem been solved?