[OBS] Error to authenticate to backend S3 on OBS using federated AK and SK

[brunoagueda]

Terraform provider version

bforghie@uvairz0120:~terraform> terraform -v
Terraform v1.4.6
on linux_amd64

Affected Resource(s)

terraform {
  backend "s3" { } 
}

Terraform Configuration Files

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key.

terraform {
  required_providers {
    opentelekomcloud = {
      source = "opentelekomcloud/opentelekomcloud"
      version = ">=1.35.4"
    }
  }
}

provider "opentelekomcloud" {
  token          = var.otc_access_token
  domain_name    = var.domain_name
  tenant_name    = var.tenant_name
  auth_url       = var.url_iam
}
terraform {
  backend "s3" {
    endpoint = "https://obs.eu-ch2.sc.otc.t-systems.com"
    bucket = "test"
    key    = "test"
    region = "eu-ch2"
    skip_region_validation      = true
    skip_credentials_validation = true
    skip_metadata_api_check     = true
    force_path_style            = true
  }
}

Debug Output/Panic Output

https://gist.github.com/brunoagueda/1156e9593bc8935109b9b3e4d5bc7c93

Steps to Reproduce

terraform init -backend-config="access_key=$ak" -backend-config="secret_key=$sk"

Expected Behavior

Terraform should have accessed the OBS bucket to store/access the state file

Actual Behavior

Error message: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.

Important Factoids

I'm authenticating with an Azure AD account on Openstack CLI, generating a token, AK and SK and using this credentials on terraform. The token works to create resources with terraform, the problem is authenticating to the backend bucket with the AK and SK

References

[artem-lifshits]

Hi there @brunoagueda. I wasn't able to reproduce the issue, are you sure you are using the right AK/SK?

Output from my side:

test-project % terraform init -backend-config="access_key=$ak" -backend-config="secret_key=$sk"

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...
- Finding opentelekomcloud/opentelekomcloud versions matching "~> 1.35.5"...
- Installing opentelekomcloud/opentelekomcloud v1.35.5...
- Installed opentelekomcloud/opentelekomcloud v1.35.5 (self-signed, key ID 3EDA0171114F71DF)

Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

[brunoagueda]

Hi @artem-lifshits I'm authenticating on the OTC console with a federated user (Azure AD user) and generating the AK and SK in My Credentials > Temporary Access Key > Generate I also generated AK and SK using the api (https://iam-pub.eu-ch2.sc.otc.t-systems.com/v3.0/OS-CREDENTIAL/securitytokens) but it doesn't work either. When I generate AK and SK with a IAM user, it works, the problem is just with a federated user account.

[artem-lifshits]

@brunoagueda you have to provide security_token for temporary ak/sk. Backend initialisation command will look like this:

terraform init -backend-config="access_key=$ak" -backend-config="secret_key=$sk" -backend-config="token=$security_token"

[brunoagueda]

@artem-lifshits tried that but got the same result

bforghie@uvairz0120:~/terraform_projects/otc_lab> terraform init -backend-config="access_key=$ak" -backend-config="secret_key=$sk" -backend-config="token=$token"

Initializing the backend...
Error refreshing state: InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
        status code: 403, request id: 0000018A17865F548145E314917B39C6, host id: 32AAAQAAEAABAAAQAAEAABAAAQAAEAABCSWAy1U9W83lwKKYSKwf40py9/QeeT/w

[artem-lifshits]

@brunoagueda that's strange, for me both permanent and temporary ak/sk worked fine. Which tenant name and auth url are you using?

[brunoagueda]

tenant name: "eu-ch2" auth_url: "https://iam-pub.eu-ch2.sc.otc.t-systems.com/v3"

[artem-lifshits]

Can you try to reinitialise backend with terraform init -migrate-state and try again? I got the same error once with old backend setting but running this command and then initialising via ak/sk/token worked for me.

[brunoagueda]

Still the same error

bforghie@uvairz0120:~/terraform_projects/otc_lab> terraform init -migrate-state -backend-config="access_key=$ak" -backend-config="secret_key=$sk" -backend-config="token=$token" 

Initializing the backend...
Backend configuration changed!

Terraform has detected that the configuration specified for the backend
has changed. Terraform will now check for existing state in the backends.

╷
│ Error: Error loading state:
│     InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
│       status code: 403, request id: 0000018A1D66B3DB8144C08198EE66FC, host id: 32AAAQAAEAABAAAQAAEAABAAAQAAEAABCSve2gB5zfUd7oq9o3CzuSJ0RVH8s5uC
│ 
│ Terraform failed to load the default state from the "s3" backend.
│ State migration cannot occur unless the state can be loaded. Backend
│ modification and state migration has been aborted. The state in both the
│ source and the destination remain unmodified. Please resolve the
│ above error and try again.
│ 
│ 

[brunoagueda]

Hi @artem-lifshits

I tried now with bash terraform init -reconfigure and it worked.

May I suggest adding this backend config to the documentation? It's not very clear in the documentation that I should use AK, SK and the STS token together when authenticating with a federated user.

Thank you very much for the help!

[artem-lifshits]

Documentation will be updated on next release.