Unable to use terraform init output as artefact for later jobs

[rdvansloten]

Describe the bug When running a pipeline with a big chain of Terraform using the same providers and modules, Terraform init takes > 30s per run. The more modules of providers we have, the longer this takes.

We have put the init files (.terraform folder, lock file) into an artifact and pass this between steps. Our pipeline runs validate, plan, apply, destroy in different stages, but they all require init. Even with the init files present, simply re-running the init step takes 10-15s per step.

When omitting the init step before running plan/destroy/apply, the pipeline throws an error:

Error: Backend initialization required, please run "terraform init"
│ 
│ Reason: Unsetting the previously set backend "azurerm"

I have noticed that when passing all the backendAzureSomething args to the init step, it runs az login before commencing, which then allows the entire stage to use that session. Passing these args to the plan step does nothing (makes sense, in the source code there is no backend init code for the plan step in TerraformCLI)

Is there any way to connect the backend into the flow without running terraform init? We run Terraform so often and cutting out the init from every stage would reduce our runs by at least 30-60 seconds, saving hours per week.

We have gotten this to work when not using this AzDo plugin, but we would rather use it.

To Reproduce Steps to reproduce the behavior:

1. Remove init from the steps
2. Execute pipeline
3. See error

Expected behavior I want to run init once, pass the generated files to the plan/apply/destroy steps and skip running terraform init again. This works when I run Terraform through the Azure CLI V2 task. Example below. Artifact is generated in a separate stage/build, then download in the apply phase:

      - task: DownloadPipelineArtifact@2
        displayName: Download plan file
        inputs:
          source: current
          artifact: tfPlan$(Build.BuildId)
          path: ${{ parameters.workingDirectory }}

      - task: DownloadPipelineArtifact@2
        displayName: Download provider files
        inputs:
          source: current
          artifact: tfInit$(Build.BuildId)
          path: ${{ parameters.workingDirectory }}

      - task: Bash@3
        displayName: 'Unzip provider files'
        inputs:
          targetType: inline
          workingDirectory: ${{ parameters.workingDirectory }}
          script: |
            unzip init.zip

      - task: AzureCLI@2
        displayName: 'Terraform Apply'
        inputs:
          scriptType: bash
          scriptLocation: inlineScript
          azureSubscription: 'AzureRM'
          addSpnToEnvironment: true
          workingDirectory: ${{ parameters.workingDirectory }}
          inlineScript: |
            export ARM_CLIENT_ID=$servicePrincipalId
            export ARM_CLIENT_SECRET=$servicePrincipalKey
            export ARM_SUBSCRIPTION_ID=$(az account show --query id | xargs)
            export ARM_TENANT_ID=$(az account show --query tenantId | xargs)
            echo "Authorizing to Azure Subscription $(az account show --query id | xargs) with Service Principal $servicePrincipalId"

            terraform apply planfile

Pipeline Logs Include logs that help demonstrate the problem. Please make sure to redact any sensitive info such as secrets.

│ Error: Backend initialization required, please run "terraform init"
│ 
│ Reason: Unsetting the previously set backend "azurerm"
│ 
│ The "backend" is the interface that Terraform uses to store state,
│ perform operations, etc. If this message is showing up, it means that the
│ Terraform configuration you're using is using a custom configuration for
│ the Terraform backend.
│ 
│ Changes to backend configurations require reinitialization. This allows
│ Terraform to set up the new configuration, copy existing state, etc. Please
│ run
│ "terraform init" with either the "-reconfigure" or "-migrate-state" flags
│ to
│ use the current configuration.
│ 
│ If the change reason above is incorrect, please verify your configuration
│ hasn't changed and try again. At this point, no changes to your existing
│ configuration or state have been made.

Agent Configuration

• OS: Ubuntu
• Hosted Agent
• Terraform 1.3.7
• AzureCLI version used: Default

[jason-johnson]

So I assume the pipeline you're having issues with looks the same as the provided one except the last task is with task CLI apply?

[rdvansloten]

@jason-johnson I am not sure what you are referring to, but what we run is basically this:

steps:
  - task: TerraformCLI@0
    displayName: 'Terraform : init'
    inputs:
      command: init
      backendType: ${{ parameters.backendType }}
      workingDirectory: ${{ parameters.workingDirectory }}
      backendServiceArm: ${{ parameters.backendServiceArm }}
      backendAzureRmSubscriptionId: ${{ parameters.backendAzureRmSubscriptionId }}
      ensureBackend: true
      backendAzureRmResourceGroupName: ${{ parameters.backendAzureRmResourceGroupName }}
      backendAzureRmStorageAccountName: ${{ parameters.backendAzureRmStorageAccountName }}
      backendAzureRmResourceGroupLocation: ${{ parameters.backendAzureRmResourceGroupLocation }}
      backendAzureRmStorageAccountSku: ${{ parameters.backendAzureRmStorageAccountSku }}
      backendAzureRmContainerName: ${{ parameters.backendAzureRmContainerName }}
      backendAzureRmKey: ${{ parameters.backendAzureRmKey }}

steps:
  - task: TerraformCLI@0
    displayName: 'Terraform : plan'
    inputs:
      command: plan
      workingDirectory: ${{ parameters.workingDirectory }}
      publishPlanResults: ${{ parameters.publishPlanResults }}
      environmentServiceName: ${{ parameters.environmentServiceName }}
      commandOptions: ${{ parameters.commandOptions }} ${{ parameters.additionalParameters }}

steps:
  - task: TerraformCLI@0
    displayName: 'Terraform : apply'
    inputs:
      command: apply
      workingDirectory: ${{ parameters.workingDirectory }}
      environmentServiceName: ${{ parameters.environmentServiceName }}
      commandOptions: ${{ parameters.commandOptions }} ${{ parameters.additionalParameters }}
      providerAzureRmSubscriptionId: ${{ parameters.providerAzureRmSubscriptionId }}

These snippets are called in different stages via -template:. Without calling the init snippet at the start of every stage, the error occurs.

[jason-johnson]

Ok, we'll have to reproduce this. I'm not sure why it would work with AzureCLI and not here.

[jason-johnson]

Can you inspect the artefacts when doing init with AzureCLI versus doing the same init with TerraformCLI? Since terraform is seeing the backend files but thinks they are different, the issue is almost certainly similar to this one.

I must say, this is also a very different strategy of using terraform that I've not seen or heard of before. What is the reason for doing this? Are you trying to cache the downloading of providers to apply them on many different and unrelated terraform environments?

[rdvansloten]

Hi @jason-johnson , you've linked this thread in your comment.

The reason we're doing this is to pass all the downloaded providers and modules to the next stage. Running an init during each stage takes quite a long time and costs several 100MB of bandwidth each time we init, we sought to optimize this.

[piizei]

This is the correct link: Stackoverflow - TF backend initialization Could you inspect in the pipeline artefacts if they contain azure-subscription-id in the .tfstate file? The az-cli task would work in this situation because it has ARM_SUBSCRIPTION_ID set.

[jason-johnson]

Hi @rdvansloten, did you see piizei's comment?

[jason-johnson]

@rdvansloten , any update?

[jason-johnson]

No update in a month, closing. If the issue still persists and the above didn't help, feel free to re-open.