Lack of Terraform S3(ObjectStorage) Backend Documentation

[robertadesso1]

We plan on deploying our infrastructure via Terraform and want to use a StackIt Bucket as backend for our .tfstate. There is a lack of documentation regarding this topic in the terraform registry or the StackIt.

Our current configuration looks like this:

terraform {
  required_version = ">=1.6.3"
  backend "s3" {
    bucket = "tfstaterobert"
    key      = "terraform.tfstate"
    endpoints = {
      s3 = "https://object.storage.eu01.onstackit.cloud"
    }
    skip_region_validation = true
    region = "eu01"
    skip_credentials_validation = true
    skip_requesting_account_id = true
    secret_key = "<access key>"
    access_key = "<access key id>"
  }

The access_key is the access key id out of the buckets credentials out of the stackit web console. The secret_key is the access key out of the buckets credentials out of the stackit web console.

When trying to relocate an already existing .tfstate to the bucket we get this error:

╷
│ Error: Error copying state from the previous "local" backend to the newly configured
│ "s3" backend:
│     failed to upload state: operation error S3: PutObject, https response error StatusCode: 400, RequestID: 1702395940886302, HostID: 12237575, api error InvalidArgument: Invalid Argument
│
│ The state in the previous backend remains intact and unmodified. Please resolve
│ the error above and try again.
│
│
╵

with

2023-12-12T16:45:50.858+0100 [DEBUG] backend-s3: HTTP Response Received: aws.region=eu01 aws.s3.bucket=tfstaterobert aws.s3.key=terraform.tfstate rpc.method=PutObject rpc.service=S3 rpc.system=aws-api tf_aws.custom_endpoint=true tf_aws.sdk=aws-sdk-go-v2 tf_backend.operation=Put tf_backend.req_id=d839bf67-80d7-4521-6b31-9e7434270aaf tf_backend.s3.bucket=tfstaterobert tf_backend.s3.path=terraform.tfstate http.duration=25 http.status_code=400 http.response_content_length=210 http.response.header.date="Tue, 12 Dec 2023 15:45:40 GMT" http.response.header.server=StorageGRID/11.6.0.12
  http.response.body=
  | <?xml version="1.0" encoding="UTF-8"?>
  | <Error><Code>InvalidArgument</Code><Message>Invalid Argument</Message><Resource>/terraform.tfstate?x-id=PutObject</Resource><RequestId>1702395940886302</RequestId></Error>
   http.response.header.x_ntap_sg_trace_id=905b2d83a2aed5f http.response.header.content_type=application/xml http.response.header.connection=keep-alive http.response.header.x_amz_request_id=1702395940886302 http.response.header.x_amz_id_2=12237575
2023-12-12T16:45:50.858+0100 [DEBUG] backend-s3: request failed with unretryable error https response error StatusCode: 400, RequestID: 1702395940886302, HostID: 12237575, api error InvalidArgument: Invalid Argument: aws.region=eu01 aws.s3.bucket=tfstaterobert aws.s3.key=terraform.tfstate rpc.method=PutObject rpc.service=S3 rpc.system=aws-api tf_aws.sdk=aws-sdk-go-v2 tf_backend.operation=Put tf_backend.req_id=d839bf67-80d7-4521-6b31-9e7434270aaf tf_backend.s3.bucket=tfstaterobert tf_backend.s3.path=terraform.tfstate

When trying to create a completly new .tfstate without relocating the already existing one it looks like it was successfull, however no .tfstate is created. The credentials provided were used to connect to the bucket with the Cyberduck client as described in the StackIt documentation and are correct. I was able to create a file with the Cyberduck client with these exact credentials.

Overall there is a lack of documentation on how to get the backend configured correctly.

[vicentepinto98]

Hello @robertadesso1,

Hashicorp added a sha256 checksum validation on put operations with version 1.6.

Unfortunately we don´t support the x-amz-checksum-sha256 header on put operations.

Therefore you must skip the validation using the parameter skip_s3_checksum = true on your backend configuration to make it work with tf 1.6 onwards.

Let me know if this fixes your issue.

We will also add documentation on how to configured a terraform backend using STACKIT Object Storage.