Pre-computed plans on new terraform s3/dynamodb state configuration overwrite each other and don't get detected as stales

[rymir]

Terraform Version

Terraform v0.12.28

Terraform Configuration Files

terraform {
  required_version = "0.12.28"
  backend "s3" {
  }
}

Debug Output

terraform init -lock=true -force-copy -input=false -backend-config=bucket=XXX -backend-config=key=XXX -backend-config=region=us-east-1 -backend-config=dynamodb_table=XXX

Crash Output

no crash

Expected Behavior

1. when we run terraform init with s3 state backend configuration, we expect an s3 initial state to be written to s3
2. we then add any terraform resource and generate 2 pre-computed plans as terraform plan -out=plan1.out and terraform plan -out=plan2.out
3. when we go and apply both plans sequentially, the first one executes and succeeds, while the second one fails with a "Stale Plan" error because the underlying s3 state lineage doesn't match up

Actual Behavior

1. when we run terraform init with s3 state backend configuration, no s3 initial state is written to s3
2. we then add any terraform resource and generate 2 pre-computed plans as terraform plan -out=plan1.out and terraform plan -out=plan2.out
3. when we go and apply both plans sequentially, they both execute with the first succeeding while the second fails complaining about the resource been created already by the first
4. the state on s3 gets overwritten by the second apply and the only mitigation at this point is to "revert" the versioned s3 state to the one written by the first terraform apply, or go and manually delete the resources.

Steps to Reproduce

terraform init -lock=true -force-copy -input=false -backend-config=bucket=XXX -backend-config=key=XXX -backend-config=region=us-east-1 -backend-config=dynamodb_table=XXX
terraform plan -out=./plan1.out
terraform plan -out=./plan2.out
terraform apply ./plan1.out
terraform apply ./plan2.out

Additional Context

References

[0x91]

This is related to https://github.com/hashicorp/terraform/issues/24078

We saw that the GCS backend performs the initialisation as expected: https://github.com/hashicorp/terraform/blob/79d74c9ba0753000caafaef11700e10bcedda003/backend/remote-state/gcs/backend_state.go#L103

As there are different consistency guarantees between S3 and GCS we were unsure of the correct fix. Happy to work on a PR if there is a proposed solution.

[rymir]

FYI, our current workaround immediately after terraform init is the following:

  # on first-time tfstate creation, it prevents pre-computed plans from over-writing the s3 state when
  # the pre-computed plans have been created with no initial-state in s3 to compute a lineage on.
  # "terraform state push" might raise an error if an s3 tfstate is already in place
  # in this case we swallow the error and continue
  if [ "$(aws dynamodb get-item --table-name "${TFSTATE_DYNAMO_TABLE_NAME}" --key "{\"LockID\": {\"S\": \"${TFSTATE_S3_BUCKET_NAME}/${TFSTATE_FILE}-md5\"}}" | jq -r .Item.LockID.S)" != "${TFSTATE_S3_BUCKET_NAME}/${TFSTATE_FILE}-md5" ]; then
    echo "[INFO] No s3 tfstate detected. Ensuring an initial empty tfstate written in s3."
    terraform state push ".terraform/terraform.tfstate" || true
  fi

[jbardin]

Thank for the report and additional information @rymir! We are tracking this issue in #24078

Closing as duplicate of #24078

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.