[2022-03-16] AWS API key exposure (terraform backends)

[dduportal]

What happened?

• The 16th of March 2022 at 14:46 UTC, some of the jenkins-infra AWS APi keys were accidentally exposed in a GitHub status check (through Jenkins console output).
• At 14:49 UTC the exposed keys were disabled and removed.
• Root cause is the pull request https://github.com/jenkins-infra/azure/pull/174 which goal is to start terraform management of azure resource on a new empty project with a new backend.
  • The commit https://github.com/jenkins-infra/azure/pull/174/commits/26a4f497a0bfe01da29b6bd302ab661264e2623d forgot to update the credential ID to use azure terraform credentials instead of aws terraform credentials. Classic case of "bad copy and paste".
  • When running the command terraform init -backend-config=<secret file> with the secret file having an incorrect (syntax error) content, its content was exposed by terraform on the stderr.
  • This pipeline job uses a GitHub Branch source which as the GitHub Status Check enabled: the console output was forwarded to GitHub Checks page leading to the exposure.

What could be the impacts?

• The exposed AWS API keys are used by Terraform to access its backend config in S3 buckets. As they are restricted to only the S3 service + only to their respective buckets, there was not any impact on the AWS resources.
• The audit trail logs showed that the exposed keys were not used since 14:46 UTC. The content of the S3 buckets was not changed since 14:42 UTC so there was no impact.
• The Jenkinsfile(_k8s) changed in the pull request can only be executed when a Jenkins infra administrator change it. An external contributor opening a pull request would not have resulted in this behavior: infra.ci.jenkins.io is configured to use the Jenkinsfile of the principal branch when a non-admin open a pull request.

How to avoid this from happening again?

Exchanges with different community members, through public and private channels, led to the following list of counter measure to avoid this problem to happen:

Short Term

What can we do right now?

• [x] Of course fix the PR so that it uses the correct credentials

• [x] Change the GitHub status check configuration on infra.ci.jenkins.io

  • Ideally: decrease the verbosity to avoid Terraform leaking sensitive data by error when failing
  • Fallback: disable github status check (only keep github status notification) on infra.ci.jenkins.io
  • Done:
  • https://github.com/jenkins-infra/kubernetes-management/pull/2103
  • https://github.com/jenkins-infra/kubernetes-management/pull/2107

• [x] #2841

• [x] #2840

• [x] Slapping my face for such a dumb copy-and-paste error.

• [x] Regenerate the AWS API keys

Long Term

What should be think about to fix this in a better way?

• Avoid usage of file credential in Jenkins: the "string"-like usual credentials are automatically hidden by Jenkins in the console output

  • Challenge: It means WAY more credentials to manage if we do 1:1 mapping (either in Jenkins and for the terraform init command)

• Move the credentials to a Vault/KMs and delegate the authorization to this system (instead of Jenkins)

  • Challenge: would mean start something from scratch, but would allow a lot of security features (automatic rotation, centralized credentials, etc.)

• Use the Terraform Cloud Saas to store our backends

  • There is an Open-Source free plan (with limits)
  • Might be an opportunity to partnership with Hashicorp
  • Would simplify the credentials in Jenkins

Links

• Pull Request https://github.com/jenkins-infra/azure/pull/174
• The checks with the exposure: https://github.com/jenkins-infra/azure/pull/174/checks?sha=26a4f497a0bfe01da29b6bd302ab661264e2623d

[timja]

Challenge: It means WAY more credentials to manage if we do 1:1 mapping (either in Jenkins and for the terraform init command)

Could you provide an example of why this is a lot more credentials? What information you're trying to hide here?

In the case of Azure the sensitive value is the ARM_ACCESS_KEY. You can even do away with that by using a managed identity or service principal

[dduportal]

Challenge: It means WAY more credentials to manage if we do 1:1 mapping (either in Jenkins and for the terraform init command)

Could you provide an example of why this is a lot more credentials? What information you're trying to hide here?

In the case of Azure the sensitive value is the ARM_ACCESS_KEY. You can even do away with that by using a managed identity or service principal

The backend configuration have the following settings currently (all are part of the backend config)

• S3 backends (Terraform projects aws and digitalocean):

  • region
  • access_key
  • secret_key
  • bucket
  • encrypt
  • kms_key_id
  • key

• AzureRM backends (Terraform projects azure and datadog):

  • resource_group_name
  • storage_account_name
  • container_name
  • client_id
  • client_secret
  • subscription_id
  • tenant_id
  • key

[jglick]

You should not be using AWS API keys. Either use instance identity, if the whole controller can be trusted to act as a logical unit of security; or https://plugins.jenkins.io/oidc-provider/#plugin-content-accessing-aws if you need something finer-grained or are just not running the controller in AWS or on a system which can easily support AWS identities at the infrastructure level.

[timja]

which relates to https://github.com/jenkins-infra/helpdesk/issues/2922#issuecomment-1120893431 for Azure