`pulumi up` fails on self-hosted docker agent, works perfectly on hosted agent.

adriantorrie commented 4 years ago

Describe the bug pulumi up fails on self-hosted agent, works perfectly on hosted agent.

To Reproduce Steps to reproduce the behavior:

Build minimal self-hosted agent and deploy

FROM python:3.8-slim as base

ENV DEBIAN_FRONTEND=noninteractive
RUN echo "APT::Get::Assume-Yes \"true\";" > /etc/apt/apt.conf.d/90assumeyes
RUN apt-get update \
    && apt-get install -y --no-install-recommends \
        ca-certificates \
        curl \
        jq \
        git \
        iputils-ping \
        libcurl4 \
        libicu63 \
        libunwind8 \
        netcat \
        zip \
        unzip

# Required container script used at runtime with appropriate permissions
WORKDIR /azp
COPY ./src/start.sh .
RUN chmod +x start.sh

CMD ["./start.sh"]

On the self-hosted agent during a pipeline run - Install pulumi requirements

# Install pulumi and it's venv requirements in the order required for
# pulumi to work in an ADO agent.
parameters:
- name: azure_subscription
  type: string
- name: pulumi_version
  type: string
  default: "latest"
- name: stack_name
  type: string

steps:
    # Required for the infra/requirements.txt to be installed
    - checkout: self

    # Create variables required for pulumi tasks, as the pulumi
    # task runner does not respect `env` key
    - bash: |
        echo "##vso[task.setvariable variable=AZURE_STORAGE_ACCOUNT]$(ENV-PULUMI-STORAGE-ACCOUNT)"
        echo "##vso[task.setvariable variable=AZURE_STORAGE_KEY]$(ENV-PULUMI-STORAGE-KEY)"
        echo "##vso[task.setvariable variable=PULUMI_CONFIG_PASSPHRASE]${{ parameters.stack_name }}"
      displayName: export_environment_variables
      name: export_environment_variables

    # Install requirements listed in the requirements.txt 
    - bash: |
        python3 -m pip install --upgrade pip setuptools wheel \
          && pip3 install -r infra/requirements.txt
      displayName: install_pulumi_requirements
      name: install_pulumi_requirements

On the self-hosted agent during a pipeline run - Run pulumi up

# - Requires `templates/steps/pulumi_install.yml` to be run prior AND
#   within the same job/deployment.
parameters:
- name: azure_subscription
  type: string
- name: pulumi_version
  type: string
  default: "latest"
- name: stack_name
  type: string

steps:  
    # Run the pulumi deploy task 
    # - Pulumi uses golang to access blob storage
    #   https://godoc.org/gocloud.dev/blob/azureblob
    - task: Pulumi@1
      inputs:
        azureSubscription: ${{ parameters.azure_subscription }}
        command: 'up'
        loginArgs: '--cloud-url azblob://$(ENV-PULUMI-CONTAINER-NAME)'
        args: '--yes --show-reads --show-replacement-steps --show-sames --logtostderr -v=9 --non-interactive'
        stack: ${{ parameters.stack_name }}
        versionSpec: ${{ parameters.pulumi_version }}
      displayName: pulumi_up
      name: pulumi_up
      env: 
        TF_LOG: TRACE

See error

2020-10-20T04:40:38.8883704Z ##[section]Starting: pulumi_up
2020-10-20T04:40:38.9273331Z ==============================================================================
2020-10-20T04:40:38.9275276Z Task         : Pulumi Azure Pipelines Task
2020-10-20T04:40:38.9277233Z Description  : Azure Pipelines task extension for running Pulumi apps.
2020-10-20T04:40:38.9279358Z Version      : 1.***.6
2020-10-20T04:40:38.9280722Z Author       : Pulumi
2020-10-20T04:40:38.9282510Z Help         : Join us on Slack at https://slack.pulumi.com.
2020-10-20T04:40:38.9284494Z ==============================================================================
2020-10-20T04:40:41.6012635Z Downloading: https://get.pulumi.com/releases/sdk/pulumi-v2.12.***-linux-x64.tar.gz
2020-10-20T04:40:49.6697648Z Extracting archive
2020-10-20T04:40:49.7614099Z [command]/bin/tar xzC /azp/agent/_work/_temp/d2d41c37-36cb-48df-82f3-18***e3b929419 -f /azp/agent/_work/_temp/***c77f461-***d5f-429f-9***a9-8f***5143b***4c1
2020-10-20T04:40:57.9715993Z Prepending PATH environment variable with directory: /azp/agent/_work/_temp/d2d41c37-36cb-48df-82f3-18***e3b929419/pulumi
2020-10-20T04:40:58.1005104Z [command]/azp/agent/_work/_temp/d2d41c37-36cb-48df-82f3-18***e3b929419/pulumi/pulumi version
2020-10-20T04:40:59.2954896Z v2.12.***
2020-10-20T04:41:00.6333949Z 
2020-10-20T04:41:00.8680437Z [command]/azp/agent/_work/_temp/d2d41c37-36cb-48df-82f3-18***e3b929419/pulumi/pulumi login --cloud-url azblob://***
2020-10-20T04:41:01.6769223Z Logged in to sre-azure-devops-pipeline-agent-*** as root (azblob://***)
2020-10-20T04:41:01.6803335Z 
2020-10-20T04:41:01.6833708Z [command]/azp/agent/_work/_temp/d2d41c37-36cb-48df-82f3-18***e3b929419/pulumi/pulumi stack select ***
2020-10-20T04:41:02.9217304Z 
2020-10-20T04:41:02.9669584Z [command]/azp/agent/_work/_temp/d2d41c37-36cb-48df-82f3-18***e3b929419/pulumi/pulumi up --yes --show-reads --show-replacement-steps --show-sames --logtostderr -v=9 --non-interactive
2020-10-20T04:41:08.2638054Z Previewing update (***):
2020-10-20T04:41:08.2972964Z I1***2*** ***4:41:***8.296695   2665*** plugins.go:76] gatherPluginsFromProgram(): gathering plugins from language host
2020-10-20T04:41:08.2999151Z I1***2*** ***4:41:***8.299536   2665*** plugins.go:47***] GetPluginPath(language, python, <nil>): found on $PATH /azp/agent/_work/_temp/d2d41c37-36cb-48df-82f3-18***e3b929419/pulumi/pulumi-language-python
2020-10-20T04:41:08.3000984Z I1***2*** ***4:41:***8.299573   2665*** plugin.go:83] Launching plugin 'python' from '/azp/agent/_work/_temp/d2d41c37-36cb-48df-82f3-18***e3b929419/pulumi/pulumi-language-python' with args: ***:45819
2020-10-20T04:41:08.9903046Z I1***2*** ***4:41:***8.9899***8   2665*** langruntime_plugin.go:178] langhost[python].GetPluginInfo() executing
2020-10-20T04:41:09.0299614Z I1***2*** ***4:41:***9.***2565***   2665*** langruntime_plugin.go:91] langhost[python].GetRequiredPlugins(proj=***,pwd=/azp/agent/_work/1/s/infra,program=.) executing
2020-10-20T04:41:09.0490303Z I1***2*** ***4:41:***9.***48376   2665*** langruntime_plugin.go:133] langhost[python].GetRequiredPlugins(proj=***,pwd=/azp/agent/_work/1/s/infra,program=.) success: #versions=***
2020-10-20T04:41:09.0522393Z I1***2*** ***4:41:***9.***5***77   2665*** plugins.go:88] gatherPluginsFromProgram(): plugin python <nil> () is required by language host
2020-10-20T04:41:09.0525741Z I1***2*** ***4:41:***9.***51851   2665*** plugins.go:1***] gatherPluginsFromSnapshot(): gathering plugins from snapshot
2020-10-20T04:41:09.0528440Z I1***2*** ***4:41:***9.***5186***   2665*** plugins.go:1***3] gatherPluginsFromSnapshot(): no snapshot available, skipping
2020-10-20T04:41:09.0530896Z I1***2*** ***4:41:***9.***51865   2665*** plugins.go:132] ensurePluginsAreInstalled(): beginning
2020-10-20T04:41:09.0539458Z I1***2*** ***4:41:***9.***53466   2665*** plugins.go:47***] GetPluginPath(language, python, <nil>): found on $PATH /azp/agent/_work/_temp/d2d41c37-36cb-48df-82f3-18***e3b929419/pulumi/pulumi-language-python
2020-10-20T04:41:09.0542688Z I1***2*** ***4:41:***9.***53487   2665*** plugins.go:137] ensurePluginsAreInstalled(): plugin python <nil> already installed
2020-10-20T04:41:09.0546131Z I1***2*** ***4:41:***9.***53495   2665*** plugins.go:152] ensurePluginsAreInstalled(): completed
2020-10-20T04:41:09.0548852Z I1***2*** ***4:41:***9.***535***5   2665*** plugins.go:22***] computeDefaultProviderPlugins(): language host reported empty set of provider plugins, using all plugins
2020-10-20T04:41:09.0551883Z I1***2*** ***4:41:***9.***53511   2665*** plugins.go:239] computeDefaultProviderPlugins(): considering python
2020-10-20T04:41:09.0554441Z I1***2*** ***4:41:***9.***53524   2665*** plugins.go:242] computeDefaultProviderPlugins(): skipping python, not a resource provider
2020-10-20T04:41:09.0598245Z I1***2*** ***4:41:***9.***53528   2665*** plugins.go:276] computeDefaultProviderPlugins(): summary of *** plugins:
2020-10-20T04:41:09.0603488Z I1***2*** ***4:41:***9.***57419   2665*** langruntime_plugin.go:144] langhost[python].Run(pwd=/azp/agent/_work/1/s/infra,program=.,#args=***,proj=***,stack=c***,#config=***,dryrun=true) executing
2020-10-20T04:41:09.1185040Z I1***2*** ***4:41:***9.***59775   2665*** plan_executor.go:396] planExecutor.retirePendingDeletes(...): no pending deletions
2020-10-20T04:41:09.1186258Z I1***2*** ***4:41:***9.***6***473   2665*** plan_executor.go:22***] planExecutor.Execute(...): waiting for incoming events
2020-10-20T04:41:09.1188085Z I1***2*** ***4:41:***9.***6***513   2665*** step_executor.go:321] StepExecutor worker(-2): worker coming online
2020-10-20T04:41:09.1189961Z I1***2*** ***4:41:***9.***6***526   2665*** step_executor.go:321] StepExecutor worker(-2): worker waiting for incoming chains
2020-10-20T04:41:09.4849798Z 
2020-10-20T04:41:11.7689215Z I1***2*** ***4:41:11.3859***4   2665*** eventsink.go:59] registering resource: ty=pulumi:pulumi:Stack, name=***-***, custom=False, remote=False
2020-10-20T04:41:11.7691127Z I1***2*** ***4:41:11.385974   2665*** eventsink.go:62] eventSink::Debug(<{%reset%}>registering resource: ty=pulumi:pulumi:Stack, name=***-***, custom=False, remote=False<{%reset%}>)
2020-10-20T04:41:11.7692548Z I1***2*** ***4:41:11.419557   2665*** eventsink.go:59] preparing resource for RPC
2020-10-20T04:41:11.7693332Z I1***2*** ***4:41:11.419578   2665*** eventsink.go:62] eventSink::Debug(<{%reset%}>preparing resource for RPC<{%reset%}>)
2020-10-20T04:41:15.9695925Z 
2020-10-20T04:41:16.6686123Z ##[error]Pulumi command exited with code 'null' while trying to run 'up --yes --show-reads --show-replacement-steps --show-sames --logtostderr -v=9 --non-interactive'.
2020-10-20T04:41:16.8144634Z ##[section]Finishing: pulumi_up

Expected behavior

pulumi up works on self-hosted agent

Screenshots None

Additional context

The tasks above are used as-is for both self-hosted and hosted agents, they don't work on self-hosted
Base image for the Dockerfile, for the self-hosted agent, has been tried with ubuntu:18.04, the same error occurs
The following shouldn't matter, just added for completeness ...

On a self-hosted agent we have to assign the kubeconfig from Azure Key Vault, as the cluster is fire-walled and not accessible using Azure Devops Service Connections. Verification task is performed and works correctly.

      - bash: |
          mkdir -p $HOME/.kube
          cat <<< "${KUBECONFIG_LITERAL}" > $HOME/.kube/config
        env:
          KUBECONFIG_LITERAL: $(SITE_KUBECONFIG_LITERAL)
        displayName: create_kubeconfig
        name: create_kubeconfig

      - bash: |
          kubectl get namespaces
        displayName: verify_kubeconfig
        name: verify_kubeconfig

While the hosted agent uses a service connection for establishing the correct kubeconfig

      - task: Kubernetes@1
        inputs:
            connectionType: 'Kubernetes Service Connection'
            kubernetesServiceEndpoint: ${{ parameters.zone }}
            command: 'login'
        displayName: login_kubernetes
        name: login_kubernetes

praneetloke commented 4 years ago

@adriantorrie would you mind trying out the latest version v1.0.7 (released today) and see if that resolves your issue? I believe it should work as it was related to the task not setting some env vars that is used by the KeyVault secrets provider, as well as the login command not using env vars defined in the executing process.

I believe on a self-hosted agent too the task extension should auto-update itself, but you can confirm the version being used by looking at the log for the Pulumi step. The version number of the task is printed at the beginning of the step.

adriantorrie commented 4 years ago

@adriantorrie would you mind trying out the latest version v1.0.7 (released today) and see if that resolves your issue? I believe it should work as it was related to the task not setting some env vars that is used by the KeyVault secrets provider, as well as the login command not using env vars defined in the executing process.

I believe on a self-hosted agent too the task extension should auto-update itself, but you can confirm the version being used by looking at the log for the Pulumi step. The version number of the task is printed at the beginning of the step.

Will give it a go sometime this week hopefully.

praneetloke commented 3 years ago

@adriantorrie I'll close this issue but please reopen or create a new issue if you have encountered other problems with using the latest version of the task extension in your self-hosted agent.

pulumi / pulumi-az-pipelines-task

`pulumi up` fails on self-hosted docker agent, works perfectly on hosted agent. #50