Proposal: Remove credentials and access_token fields from the terraform provider.

upodroid commented 3 years ago

Affected Resource(s)

google_*

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment. If the issue is assigned to the "modular-magician" user, it is either in the process of being autogenerated, or is planned to be autogenerated soon. If the issue is assigned to a user, that user is claiming responsibility for the issue. If the issue is assigned to "hashibot", a community member has claimed the issue already.

@danawillow @rileykarson

Objective,

Removing credentials and access_token field the provider and specifying service account keys using the GOOGLE_APPLICATION_CREDENTIALS environment variable.

Background

Google Client Libraries have supported loading JSON service account keys before Google introduced the Application Default Credentials in 2015.

https://cloudplatform.googleblog.com/2015/07/Easier-Auth-for-Google-Cloud-APIs-Introducing-the-Application-Default-Credentials-feature.html

Terraform supported this since the first release https://github.com/hashicorp/terraform-provider-google/blob/v0.1.0/google/config.go#L61 https://github.com/hashicorp/terraform-provider-google/blob/v3.40.0/google/config.go#L727 However, authentication options have evolved since then and there a number of better options available.

Overview

There is a problem with having a provider configured like this:

provider "google" {
  credentials = file("~/.gce/credentials")
  project     = var.project
  region      = var.region
}

This configuration is not portable and won't work on Google Cloud. Terraform will look for that key instead of assuming the credentials of the VM or the K8s Pod/Node.
Alot of tutorials instruct people to generate a service account key instead of using User ADCs https://cloud.google.com/sdk/gcloud/reference/auth/application-default. Keys have very long livetime (10 years) and it is a bad practice in terms of security and scalability. Also, those tutorials often specify their provider configuration as shown above. https://learn.hashicorp.com/tutorials/terraform/google-cloud-platform-build
In most production uses of terraform, the credentials field is never specified but we setup the environment terraform runs on correctly. https://cloud.google.com/solutions/managing-infrastructure-as-code-with-terraform-jenkins-and-gitops and https://github.com/gruntwork-io/terraform-google-ci/tree/master/examples/cloud-build-csr-gke
Google released Service Account Impersonation which allows an identity to assume another service account.

Detailed Design

In the next release of the provider, the credentials field will be removed. Users who are using this field must specify the environment variable GOOGLE_APPLICATION_CREDENTIALS to point to the path of the key. Terraform will then evaluate the credentials in the order specified by the golang client library at https://godoc.org/golang.org/x/oauth2/google#FindDefaultCredentials which I have pasted below.

1. A JSON file whose path is specified by the
   GOOGLE_APPLICATION_CREDENTIALS environment variable.
2. A JSON file in a location known to the gcloud command-line tool.
   On Windows, this is %APPDATA%/gcloud/application_default_credentials.json.
   On other systems, $HOME/.config/gcloud/application_default_credentials.json.
3. On Google App Engine standard first generation runtimes (<= Go 1.9) it uses
   the appengine.AccessToken function.
4. On Google Compute Engine, Google App Engine standard second generation runtimes
   (>= Go 1.11), and Google App Engine flexible environment, GKE, Cloud Run it fetches
   credentials from the metadata server.

The provider docs at https://www.terraform.io/docs/providers/google/index.html will be updated to mention the following scenarios:

Running terraform locally on your workstation? Install the Google Cloud SDK and run gcloud auth application-default login
Running terraform on Google Cloud? Do nothing. Terraform will assume the service account of the VM/GKE Pod/Node. However, your VM/Nodes should have a custom service account that is scoped to cloud-platform
Running terraform outside of Google Cloud? Create a service accout key and set the GOOGLE_APPLICATION_CREDENTIALS environment variable to point to the path of the key.

The access_token field will be removed in favour of impersonate_service_account. The golang client library supports service account impersonation which replaces the need for having multiple providers and using the access_token datasource. You will need to suppply a valid Application Default Credentials and that identity must have Service Account Token Creator role on the service account you are looking to impersonate.

There is the added benefit of leaner code and simplified authentication.

~The data source google_service_account_id_token needs to be tweaked. I'm not sure why it was coded to exclude User ADCs even though it works with ADCs. Initial Commit: https://github.com/hashicorp/terraform-provider-google/pull/5670, Issue: https://github.com/hashicorp/terraform-provider-google/issues/7333~

http POST https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/terraform@REDACTED.iam.gserviceaccount.com:generateIdToken "Authorization: Bearer $(gcloud auth application-default print-access-token)" includeEmail=true audience=https://foo.com
http POST https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/terraform@REDACTED.iam.gserviceaccount.com:generateIdToken "Authorization: Bearer $(gcloud auth print-access-token)" includeEmail=true audience=https://foo.com

The data source google_storage_object_signed_url also needs work too. https://github.com/hashicorp/terraform-provider-google/issues/3558

Alternatives Considered

The alternative is to leave the credentials field but it is shame to tell newcomers that the tutorial at Hashicorp or Training Provider X is very far from what we do for real and that it has some bad practices in it.

Also, it is not uncommon to have other applications such as Packer, Ansible, __ present in the same devops/deployment pipelines which all expect Application Default Credentials so providing service account key via the environment variable shouldn't be too difficult.

rileykarson commented 3 years ago

Just a couple minor comments/thoughts from me right now. Nothing necessarily actionable- as Dana noted in https://github.com/GoogleCloudPlatform/magic-modules/pull/3999 it's likely we make a decision when we're closer to 4.0.0- just writing down some thoughts before I forget them.

Application Default Credentials only allow a single authentication method while credentials and access_token allow multiple provider blocks to be defined with distinct auth methods. Is that behaviour we don't think is being used, or shouldn't get used?
I believe the history on access_token is that users wanted it to be able to supply access tokens from Vault. Unfortunately we didn't record that in the original PR. Is that still a desirable usecase?

I would be interested if there is guidance on cloud.google.com or other sources that encourages / discourages various authentication methods. One useful precedent could be the google.golang.org/api/transport package.

@salrashid123 - you've also touched authentication/impersonation related stuff in the past, so if you've got a bit of time I'd be interested in your thoughts as well.

salrashid123 commented 3 years ago

yeah, i'm not quite sure about removing the parameters for the samereasons above (i.,e you can specify mutliple credentials at runtime).

I'm not aware of a single preferred type in the ADC hierarchy but any most situations we don't encourage handling and distributing raw secrets, if possible (you have to in a number of cases...). The ambient credentials (eg metadata) is better than having access to a file..

just for ref, there is a proposal to support "process credentials" within google libraries. In that mode, some arbitrary binary is called "that just gives" a google token. What i mean by that is it could be any binary terraform has access to wherever it runs and that itself procures the access_token. So in that case its

provider "google" {
  access_token = file("/path/to/somebinary")
  project     = var.project
}

its still a proposal but here is a type of it (to be clear, is unsupported/unofficial).

upodroid commented 3 years ago

The ADC allows you to use 1 credentials, but you can instantiate multiple providers that are impersonating different accounts if you have a niche requirements of having multiple identities running the same terraform code.
The approach of pulling access_tokens from Vault isn't a great idea. Usually, the person running terraform for GCP will have a Google account that they use to interact with the console. Therefore the authentication sources should be Google Service Accounts and Google Accounts. The users fall in to those 3 scenarios. In most cases, Vault will be configured with OIDC or LDAP auth methods for humans and platform specific auth methods(GCP/K8s/AWS/AppRole) for machines.

Another great article: https://medium.com/google-cloud/a-hitchhikers-guide-to-gcp-service-account-impersonation-in-terraform-af98853ebd37

wouterh-dev commented 3 years ago

@upodroid

The approach of pulling access_tokens from Vault isn't a great idea. Usually, the person running terraform for GCP will have a Google account that they use to interact with the console

We typically run terraform on Jenkins (outside of GCP), which is not a user. Currently we do authenticate our Jenkins instance using a service principal with Google Cloud, but instead we would like to use access tokens from Vault so that Jenkins does not have any long-lived credentials to GCP. Instead it is provisioned a temporary credential on-demand.

We can instead use Vault to generate a service_account_keys and use that, but Google imposes a limit of max 10 keys per service account, so when using that you are bound to run into this limit eventually.

So it would be very useful if the access_token functionality was retained and extended so it can read an access token from a file or a subprocess, to deal with renewal.

c2thorn commented 4 months ago

Unlikely to move forward with this in order to maintain support for OIDC.

github-actions[bot] commented 3 months ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / terraform-provider-google