Azure Storage State Provider Variables not Available to task

[JasonWhall]

Describe the bug When using the task in YAML pipelines, normally variables are declared in YAML itself e.g.

variables:
- name: AZURE_STORAGE_ACCOUNT
  value: myStorageAccount

However it seems these variables are not available to the Pulumi task at pulumi login. Instead I am presented with the error message:

error: problem logging in: unable to open bucket azblob://state: azureblob.OpenBucket: accountName is required 

To Reproduce Sample pipeline steps (Shortened for simplicity)

# Attempt setting as pipeline var
variables:
- name: AZURE_STORAGE_ACCOUNT
  value: myStorageAccount

# 2nd Attempt, setting in task
- task: Pulumi@1
  displayName: Pulumi Preview
  env:
    AZURE_STORAGE_ACCOUNT: myStorageAccount
  inputs:
    azureSubscription: $(Azure.ServiceConnection)
    command: preview
    cwd: $(pulumiWorkingDirectory)
    stack: ${{ parameters.stackName }}
    args: --diff --suppress-permalink
    loginArgs: -c azblob://$(AZURE_STORAGE_CONTAINER)

Expected behavior AZURE_STORAGE_ACCOUNT is recognised by the pulumi task and is used in the login command

[praneetloke]

@JasonWhall thank you for submitting an issue! It would really help if you could re-run your build pipeline with the system.debug variable to true as well as check the box for Enable System Diagnostics (shown below) when you run your pipeline.

Please do note that the debug mode will print any secret env vars to your build log. For that reason, please do not post the debug log here.

That should print this debug line from the Pulumi task that shows all of the environment variables available to the task. Your AZURE_STORAGE_ACCOUNT env var should be shown as well. It may be best to search the debug log since it can tend to print a lot of information when debug info is enabled.

[praneetloke]

I got a chance to put something together and see what is going on. Two things:

1. Defining variables in the config file only makes them available as variables and not environment variables.
2. There does seem to be a bug in that the login phase of the task is not reading the environment variables passed to the task using env.

You can get around the bug with the login phase by defining the variables as an agent-level variable. This is the Variables button when you are editing your pipeline.

I'll leave this issue open to fix the bug with the task-level env not being read for the login phase.

[JasonWhall]

Hi @praneetloke, thanks for picking this up. With regards to running this pipeline in Debug, the environment variable is printed out to the console, but not by the line of code you mention.

The task seems to fail before this when running the login command here

I did have a quick test with uploading a modified version of this task and when changing the loginEnvVars to include processEnv, this works as expected. However I'm now more confused than when i started 😄

• It looks like the azure-pipelines-task-lib/task library used to get the Variables should get these variables from processEnv already? So I'm not sure how this helped... Maybe this is due to how the variables are exposed to the task.

• You mentioned the variables set in YAML are not exposed as environment variables, however the documentation currently seems to suggest that they are available as environment variables? (Albeit not very clearly). This is fairly common practice to declare variables in the top of the pipeline or from a template and use them in the tasks this way, rather than inputting them into the pipeline in the UI.

• Additionally after getting past this first issue, I then ran into a problem similar to #43 as part of this as I'm using Azure KeyVault as the secrets provider. So I'm not able to proceed any further with this task at the moment. It looks like as mentioned in this issue that the go package used to authenticate to KeyVault expects a different set of variables to be set from the service connection provided,

  AZURE_CLIENT_ID
  AZURE_CLIENT_SECRET
  AZURE_TENANT_ID

This task only sets the following variables from the service connection that are used for the terraform provider.

ARM_CLIENT_ID
ARM_CLIENT_SECRET
ARM_SUBSCRIPTION_ID
ARM_TENANT_ID

[praneetloke]

@JasonWhall

The task seems to fail before this when running the login command here

Yep. This is what I realized too.

I did have a quick test with uploading a modified version of this task and when changing the loginEnvVars to include processEnv, this works as expected. However I'm now more confused than when i started

Exactly. I think this is the "fix" I mentioned in this comment.

You mentioned the variables set in YAML are not exposed as environment variables, however the documentation currently seems to suggest that they are available as environment variables? (Albeit not very clearly). This is fairly common practice to declare variables in the top of the pipeline or from a template and use them in the tasks this way, rather than inputting them into the pipeline in the UI.

Hmm I may be misled by the example snippet on that page, but I think you may be right in that they should be accessible and perhaps again the problem here is that the loginEnvVars doesn't include all of the agent's vars (i.e. processEnv).

Additionally after getting past this first issue, I then ran into a problem similar to #43 as part of this as I'm using Azure KeyVault as the secrets provider. So I'm not able to proceed any further with this task at the moment. It looks like as mentioned in this issue that the go package used to authenticate to KeyVault expects a different set of variables to be set from the service connection provided,

Ah that is unfortunate. Would you mind opening a separate issue for that? I'll get these addressed asap.

[praneetloke]

@JasonWhall the latest version 1.0.7 should fix the issues you were encountering. Both with the env vars mapped to the Pulumi task not being used by the login command, as well as the AZURE_* env vars not being set from the Service Connection. Could you please give this another try?

[JasonWhall]

Hey @praneetloke - Yes, this now allows access to the variables required and the service principal has access to the secrets provider through the AZURE_* variables by default. Thanks! 👍

For completeness I have included an example pipeline below that is known to work (providing you have the resources/connections setup)

variables:
- name: Azure.ServiceConnection
  value: myServiceConnection

- name: Azure.Storage.Account # Can also be AZURE_STORAGE_ACCOUNT
  value: myStorageAccount

- name: AZURE_STORAGE_CONTAINER # Cannot be Azure.Storage.Container
  value: myStateContainer

- name: stackname
  value: myStackName

pool:
  vmImage: ubuntu-latest

steps:
- task: AzureCLI@2
  displayName: Get Storage Account Key
  name: GetStorageKey
  inputs:
    azureSubscription: $(Azure.ServiceConnection)
    scriptType: bash
    scriptLocation: inlineScript
    inlineScript: |
      KEY=$(az storage account keys list -n $AZURE_STORAGE_ACCOUNT --query "[0].value" -o=tsv)
      echo "##vso[task.setvariable variable=STORAGE_KEY;issecret=true]$KEY"

- task: Pulumi@1
  env:
    AZURE_STORAGE_KEY: $(STORAGE_KEY) # Must be mapped in env to access a secret value
  inputs:
    azureSubscription: $(Azure.ServiceConnection)
    command: preview
    # If AZURE_STORAGE_CONTAINER not set you can pass the container in loginArgs
    # loginArgs: azblob://myStateContainer
    args: --non-interactive --diff --suppress-permalink
    stack: $(stackname)

However I've now run into #40 🤣 I will comment more info there.

[praneetloke]

@JasonWhall that's great! Thank you so much for providing the example as well. I'll go ahead and close this issue.

I saw your comment on #40. :) I'll provide my response there today.