NoCredentialProviders: no valid providers in chain error from atlantis; terraform directly works

singhswg commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Overview of the Issue

Hi! I am getting a NoCredentialProviders: no valid providers in chain error while running atlantis plan. I am running atlantis in a container and weirdly if i exec into the container and execute terraform init/plan it works.

Not sure why atlantis cannot find credentials.

Reproduction Steps

Running atlantis in a container -

sudo docker run --rm -it -p 4141:4141  -v /home/ubuntu/.aws:/home/atlantis/.aws ghcr.io/runatlantis/atlantis:v0.19.0 server --gh-user=___ --gh-token=___ --repo-allowlist="$REPO_ALLOWLIST" --gh-webhook-secret=___

Mapping local credentials file to atlantis container as shown above which has 2 profiles - default and staging

Logs

Logs from Atlantis server -

{"level":"error","ts":"2022-06-30T17:05:50.253Z","caller":"events/instrumented_project_command_runner.go:43","msg":"Error running plan operation: running \"/usr/local/bin/terraform init -input=false\" in \"/home/atlantis/.atlantis/repos/<repo-name>/3/default\": exit status 1\n\nInitializing the backend...\n╷\n│ Error: error configuring S3 Backend: no valid credential sources for S3 Backend found.\n│ \n│ Please see [[https://www.terraform.io/docs/language/settings/backends/s3.html\n│](https://www.terraform.io/docs/language/settings/backends/s3.html/n%E2%94%82)](https://www.terraform.io/docs/language/settings/backends/s3.html/n%E2%94%82%5d(https:/www.terraform.io/docs/language/settings/backends/s3.html/n%E2%94%82)) for more information about providing credentials.\n│ \n│ Error: NoCredentialProviders: no valid providers in chain. Deprecated.\n│ \tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors\n│ \n│ \n│ \n╵\n\n","json":{"repo":"<repo-name>","pull":"3"},"stacktrace":"github.com/runatlantis/atlantis/server/events.RunAndEmitStats\n

Atlantis error message -

Initializing the backend...
╷
│ Error: error configuring S3 Backend: no valid credential sources for S3 Backend found.
│ 
│ Please see https://www.terraform.io/docs/language/settings/backends/s3.html
│ for more information about providing credentials.
│ 
│ Error: NoCredentialProviders: no valid providers in chain. Deprecated.
│   For verbose messaging see aws.Config.CredentialsChainVerboseErrors
│

Environment details

Atlantis version: latest, v0.19.0
terraform version - v1.2.3

Provider.tf picking credentials like -

provider "aws" {
  region = "us-west-2"
  shared_credentials_file = "/home/atlantis/.aws/credentials"
  profile                  = "staging"
}

backend.tf picking credentials like -

terraform {
  backend "s3" {
   ...
    shared_credentials_file = "/home/atlantis/.aws/credentials"
    profile                  = "staging"
  }
}

Any other information you can provide about the environment/deployment.

atlantis plan fails
terraform plan from container directly works

bash-5.1# terraform init

Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

singhswg commented 2 years ago

I am not specifying a separate atlantis.yml in the repo

singhswg commented 2 years ago

Test specifying atlantis.yaml as was talked here - https://github.com/runatlantis/atlantis/issues/1204, and passed repos.yaml to server -

It did not work either

#version: 3
#projects:
#- name: test-workflow
#  dir: .
#  workflow: flow1
#workflows:
#  flow1:
#    plan:
#      steps:
#      - env:
#           name: AWS_PROFILE
#           value: staging
#      - init
#      - plan
#    apply:
#      steps:
#      - env:
#           name: AWS_PROFILE
#           value: staging
#      - plan
#      - apply

andyshinn commented 2 years ago

This might be due to the providers being locked for one arch but Atlantis running as another. I usually lock for all providers. Try locking for all arches and then committing the updated lockfile:

terraform providers lock -platform=windows_amd64 -platform=darwin_amd64 -platform=linux_amd64 -platform=darwin_arm64

runatlantis / atlantis