Error Authenticating to GCP in TerraformTaskV1

[dvmorris]

I am able to use TerraformTaskV1 with a GCP Service Connection successfully, but when I also have a set of gcloud commands in my pipeline that also establishes an authenticated connection to GCP, the TerraformTaskV1 no longer authenticates in subsequent build tasks.

Download Secure File (GCP Service Account JSON Key File)

bash Task Step

gcloud auth activate-service-account --key-file=/home/vsts/work/_temp/service-account-key.json

Terraform Task Output

2020-03-26T19:06:35.4968455Z [command]/opt/hostedtoolcache/terraform/0.12.23/x64/terraform validate
2020-03-26T19:06:36.2344747Z Success! The configuration is valid.
2020-03-26T19:06:36.2346081Z 
2020-03-26T19:06:36.2454409Z [command]/opt/hostedtoolcache/terraform/0.12.23/x64/terraform apply -auto-approve tfplan
2020-03-26T19:06:36.2770148Z 
2020-03-26T19:06:36.2774259Z Error: Error parsing credentials '/home/vsts/work/r1/a/credentials-8aad8d24-69bd-48f0-b588-39275a8a8361.json': invalid character '/' looking for beginning of value
2020-03-26T19:06:36.2775281Z 
2020-03-26T19:06:36.2776318Z 
2020-03-26T19:06:36.2868375Z ##[error]Error: The process '/opt/hostedtoolcache/terraform/0.12.23/x64/terraform' failed with exit code 1

To reiterate, the TerraformTaskV1 works fine when I don't have any bash scripts that called gcloud auth activate-service-account ..., and it only stops working when I add in this step before the TerraformTaskV1 task.

Somehow the service account key file that is generated in this code: https://github.com/microsoft/azure-pipelines-extensions/blob/master/Extensions/Terraform/Src/Tasks/TerraformTaskV1/src/gcp-terraform-command-handler.ts#L23-L24

is corrupted somehow by the introduction of this gcloud authentication step.

[Romiko]

Hi,

I am having the same problem. I am using multistage pipelines and the second time I run init it fails.

Terraform init works in my validate step. When I try run init again in the plan step it fails with the same errors.

` Starting: Terraform initialize

Task : Terraform Description : Execute terraform commands to manage resources on AzureRM, Amazon Web Services(AWS) and Google Cloud Platform(GCP) Version : 0.0.142 Author : Microsoft Corporation Help : Learn more about this task

/usr/local/bin/terraform init -backend-config=bucket=terraform_state -backend-config=prefix=dev -backend-config=credentials=/home/vsts/work/1/s/credentials-cfa5ddfd-8e2c-40bc-80b2-905615f69578.json Initializing modules...

Initializing the backend... Backend configuration changed!

Terraform has detected that the configuration specified for the backend has changed. Terraform will now check for existing state in the backends.

Error: Error parsing credentials '/home/vsts/work/1/s/credentials-cfa5ddfd-8e2c-40bc-80b2-905615f69578..json': invalid character '/' looking for beginning of value

[error]Error: The process '/usr/local/bin/terraform' failed with exit code 1

Finishing: Terraform initialize `

This is the structure of the pipeline.

` trigger: batch: true branches: include:

• master

stages:

• stage: Validate displayName: Validate Terraform variables:

  • name: templatePath value: src/terraform/resources

  • name: artifacts value: SC-Infrastructure-artifacts jobs:

  • job: Validate pool: vm: ubuntu-latest workspace: clean: all steps:

  • template: pipelines/terraform-action.yaml
    parameters: environment: dev region: aae action: validate workingDirectory: $(templatePath)

  • publish: src/terraform artifact: $(artifacts)

• template: pipelines/terraform-plan.yaml parameters: name: Plan_DEV_AAE displayName: Plan DEV AAE dependsOn: Validate environment: dev region: aae `

So the step/action validate works. But when the subpipeline gets called (terraform-plan.yaml), it fails.

` parameters:

• name: name type: string
• name: displayName type: string
• name: dependsOn type: object
• name: environment type: string
• name: region type: string
• name: artifact type: string default: SC-Infrastructure-artifacts
• name: workingDirectory type: string default: $(Pipeline.Workspace)/SC-Infrastructure-artifacts/resources

stages:

• stage: ${{ parameters.name }} displayName: ${{ parameters.displayName }} dependsOn: ${{ parameters.dependsOn }} jobs:

  • job: Plan pool: vm: ubuntu-latest workspace: clean: all steps:

  • download: current artifact: ${{ parameters.artifact }}

  • template: terraform-action.yaml
    parameters: environment: ${{ parameters.environment }} region: ${{ parameters.region }} action: plan workingDirectory: ${{ parameters.workingDirectory }} `

terraform-action.yaml ` parameters:

• name: environment type: string
• name: region type: string
• name: action type: string
• name: workingDirectory type: string default: $(Pipeline.Workspace)/SC-Infrastructure-artifacts/resources

steps:

• task: ms-devlabs.custom-terraform-tasks.custom-terraform-release-task.TerraformTaskV1@0 displayName: 'Terraform initialize' inputs: provider: gcp backendServiceGCP: 'gcp-dev-rom' backendGCPBucketName: 'terraform_state' backendGCPPrefix: '${{ parameters.environment }}'
  workingDirectory: "${{ parameters.workingDirectory }}"

• bash: | chmod -R 755 '${{ parameters.workingDirectory }}' displayName: fix file permissions

• task: ms-devlabs.custom-terraform-tasks.custom-terraform-release-task.TerraformTaskV1@0 displayName: "Terraform ${{ parameters.action }}" inputs: provider: gcp

  commands validate, plan, apply, destroy

  command: ${{ parameters.action }} ${{ if or(eq(parameters.action, 'plan'), eq(parameters.action, 'apply'), eq(parameters.action, 'destroy')) }}: environmentServiceNameGCP: 'gcp-${{ parameters.environment }}-rom' backendServiceGCP: 'gcp-${{ parameters.environment }}-rom' backendGCPBucketName: 'rom_terraform_state' backendGCPPrefix: '${{ parameters.environment }}'
  workingDirectory: "${{ parameters.workingDirectory }}" ${{ if not(eq(parameters.action, 'validate')) }}: commandOptions: '-var-file="${{ parameters.environment }}.tfvars" -var "region=${{ parameters.region }}"' `

[Romiko]

I can get it working if I remove multi-stage pipelines. Is it possible you can check if this extension is compatible with multistage pipelines? I think when you run multiple terraform inits, the terraform detects a backend change and wants to reinit and then cannot read the credential file.

So multistage pipelines seems to cause this.