dmaltsiniotis
commented
6 months ago @dmaltsiniotis please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.
@microsoft-github-policy-service agree [company="{your company}"]Options:
- (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.
@microsoft-github-policy-service agree
- (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.
@microsoft-github-policy-service agree company="Microsoft"Contributor License Agreement
@microsoft-github-policy-service agree company="Demetri Maltsiniotis, LLC"
mericstam
richbjhudson
This PR attempts to address issue #145.
Problem
The behavior of the Terraform Google provider changed after the release of 4.0.0, and this change is slightly incompatible with the way the
TerraformCommandHandlerGCPclass handles the generation of Google Cloud specific 'credentials' files used for authorizing Terraform to a Google Cloud project (akin to a Service Principle in Azure nomenclature). Specifically, the issue arises during the use of multi-stage pipelines.Analysis
Historically, when a Terraform plan was saved with the -out parameter, it also embedded a file path to the credential-files.json containing the authorization credentials to the Google project. However, if the credentials-file.json was available via environment variable, the order of precedence would favor environment variable over the embedded path in the plan file. So long as the credential file existed in the "GOOGLE_CREDENTIALS" environment variable, everything would work just fine because this task would generate a new credential file on every stage/job, and set the variable accordingly.
When v4.0.0.0 was published, this order of precedence changed. Now, instead favoring the environment variable "GOOGLE_CREDENTIALS" first, the provider favors the embedded file path first. The reason this becomes an issue in multi-stage pipelines is because of the use of uuid4() to generate the credentials file:
This results in a randomly named file saved to disk. The filename that was used to generate the plan in the first stage, no longer exists on the second/apply stage, since each stage is not guaranteed to be run on the same ephemeral Azure DevOps agent. This will manifest itself is a few possible error message including:
Solution
Approach
There were a few different proposed solutions to the issue, and I ended up going with the most straightforward change that had the least impact. My goals was to have the task generate a credential filename that was the same for any given build/service connection, so that even between stages if the credential file was re-generated the embedded path in the plan file would remain valid.
My initial thought was to use some kind of MD5 or SHA-1 hashing, and hash a known value to generate the file name. After a quick analysis of the code though, I did not want to introduce any new dependencies as part of this change such as including NodeJS's 'crytpo' or other potentially bloated hashing NPM packages. Luckily, the functionality I was looking for was staring me right in the face: The already included UUID package.
Some quick history on UUID's:
The actual change
This PR updates the jsonKeyFilePath property to instead use
uuidV5()instead ofuuidV4()This results in a filename that is the deterministic based on the service connection name, and safe for writing to disk on any platform/OS:In this case, the value we are hashing is the serviceName variable. This could just have easily been build ID or anything else that remains constant between stages of a pipeline run that the task has access to.
The namespace (yet another UUID) I chose for uuidV5() is
uuidV5.URL, which is a published, known, constant generally used to represent URLs (6ba7b811-9dad-11d1-80b4-00c04fd430c8). There are four preexisting namespace hard-coded GUIDs defined by RFC 4122, in this case it doesn't matter what the namespace is, so long as it is consistent between stages, so I elected to use somewhat friendly lookinguuidV5.URLrather than make up some arbitrary UUID value in-line:Gross, right? But I digress. I made sure to test this all thoroughly with the version of UUID in actual use by the package lock file: 3.4.0.
Testing
This took some time and effort to test. My testing methodology consisted of:
Verifying all the unit tests in the projects run (by following the CONTRIBUTING.md guide)
Publishing a self/dev version of the Task to the private marketplace (again per the guide)
Creating a new DevOps organization
Installing the self/dev VSTX package into the organization
Creating a new AzDo Project, repository, pipeline
Creating a new multi-stage deployment YAML file with some Hello World! Terraform that would publish and download a plan file
Creating a new Google Cloud project, state file bucket, service account, and service account key
Adding the service account to the AzDo organization as a GCP Terraform service account
Kicking off the pipeline, observe success.
Remove self/dev VSTX, install marketplace Terraform tasks
Re-run the pipeline from the same branch/hash/commit, observe failure.
Re-run the pipeline steps above on both:
Evidence
First, using the current marketplace version:
Pipeline runs, fist is marketplace task, second is task from my branch/PR:
In the current/failing task, notice the UUID is different for the generated credential files between the plan and apply stages. Plan:
Apply:
Now, using the VSTX from this PR:
Using the Terraform task from this PR, we can observe that UUID for the credentials files are now consistent between stages: Plan:
Apply:
Thank you for your consideration.