Project factory assumes credentials.json

[mengesb]

While I understand that Terraform Enterprise is a minority, it would be nice to not rely on the content and location of an explicit file. In the case of Terraform Enterprise, and also other automation tools like Jenkins and the like, an ephemeral container is spawned with a limited toolset and no common local variables and files (such as ~/.some_hidden_directory).

For the google provider, it's simple enough to generate a map variable and provide that using the jsonencode() terraform interpolation:

variable "gcloud" {
  type = "map"

  default = {
    type                        = ""
    project_id                  = ""
    private_key_id              = ""
    private_key                 = ""
    client_email                = ""
    client_id                   = ""
    auth_uri                    = ""
    token_uri                   = ""
    auth_provider_x509_cert_url = ""
    client_x509_cert_url        = ""
  }
}

variable "google" {
  default = {
    project = ""
    region  = ""
    zone    = ""
  }
}

provider "google" {
  version     = "~> 2.0"
  credentials = "${jsonencode(var.gcloud)}"
  project     = "${var.google["project"]}"
  region      = "${var.google["region"]}"
  zone        = "${var.google["zone"]}"
}

provider "google-beta" {
  version     = "~> 2.0"
  credentials = "${jsonencode(var.gcloud)}"
  project     = "${var.google["project"]}"
  region      = "${var.google["region"]}"
  zone        = "${var.google["zone"]}"
}

This removes the need to have the typical credentials.json file for the provider, however in the case of the preconditions.py, this file is still in fact needed. If instead there was a method or way also use variables in place of the file, this would allow portability and security, because what is typically in that json file can be provided as a git ignored mypersonal.auto.tfvars file and instructions for the user to generate one. In the Terraform Enterprise case, you add sensitive/protected variables via the UI or API that result in a temporary terraform.tfvars file that lives only with the container; cleaned up once the job completes, with or without failure.

[morgante]

@mengesb We don't require the credentials.json file be provided, actually.

The module (including preconditions.py) will automatically use Application Default Credentials if they are available.

In Terraform Enterprise, is it not possible to set environment variables?

[mengesb]

@morgante my intrepretations of application default credentials is that the env var points to the json file... Am I missing something other than this segment:

export GOOGLE_APPLICATION_CREDENTIALS="/home/user/Downloads/[FILE_NAME].json"

[aaron-lane]

@mengesb I believe setting GOOGLE_CREDENTIALS to the contents of a key file should suffice.

[mengesb]

@aaron-lane there's quite a bit that's in the credentials.json file, which value of the following keys are you referring to? My code snippet above shows all the keys present in the json file, are you referring to the value for private_key ?

[mengesb]

I tried exporting the private_key value as GOOGLE_CREDENTIALS, pTFE still gives me this response on the preconditions:

module.host-project.module.project-factory.null_resource.preconditions: Provisioning with 'local-exec'...
module.host-project.module.project-factory.null_resource.preconditions (local-exec): Executing: ["/bin/sh" "-c" "/terraform/g-terraform-svpc-host-project-repo/.terraform/modules/02e55eefdd476a220d5e915a4816eee1/scripts/preconditions.sh \\\n    --credentials_path '' \\\n    --billing_account '<REDACTED>' \\\n    --org_id '<REDACTED>' \\\n    --folder_id '<REDACTED>' \\\n    --shared_vpc ''\n"]
module.host-project.module.project-factory.null_resource.preconditions (local-exec): Unable to import Google API dependencies, skipping GCP precondition checks!
module.host-project.module.project-factory.null_resource.preconditions: Creation complete after 0s (ID: 473800249239451708)

...

* google_project.main: Error setting billing account "<REDACTED>" for project "<REDACTED>": googleapi: Error 403: Cloud Billing API has not been used in project <REDACTED> before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudbilling.googleapis.com/overview?project=<REDACTED> then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry., accessNotConfigured

[mengesb]

So all in all this has been an enlightening experience. I can get around the credentials_path requirement by creating a var above as described, and then using the local_file resource to write this out to the file system on the pTFE container. the trigger of course would also have to update and fire on resources where this is necessary. I'm working through an operational example currently, just recently got setup-sa.sh to finally execute correctly so I'm busy cleaning up after all those failures and will resume working with the project factory on the new SA.

[aaron-lane]

@mengesb what I meant was to read the contents of credentials.json in to GOOGLE_CREDENTIALS, like export GOOGLE_CREDENTIALS="$(< credentials.json)".

[morgante]

I'm going to go ahead and close this because Project Factory should work with credentials provided from the environment as @aaron-lane described. If you run into further blockers, please reopen.