A better way to install plugins

[darnmason]

I use a build pipeline to publish an artifact of my Pulumi project and another pipeline downloads that artifact to execute it.

This means that I have Pulumi configured with the binary option:

name: project-name
runtime:
  name: dotnet
  options:
    binary: Project.Name.dll
description: Blah

I discovered that this means Pulumi can't/won't install plugins used by the binary. So now I've resorted to the following steps when executing Pulumi:

- task: Pulumi@1
  displayName: 'Install Azure-Native'
  inputs:
    azureSubscription: '$(azureSubscription)'
    command: plugin
    args: 'install resource azure-native v0.7.1'
    cwd: '$(workingDir)'
    stack: '$(pulumiStack)'
    createStack: true
- task: Pulumi@1
  displayName: 'Install Random'
  inputs:
    azureSubscription: '$(azureSubscription)'
    command: plugin
    args: 'install resource random v3.1.0'
    cwd: '$(workingDir)'
    stack: '$(pulumiStack)'
    createStack: true
- task: Pulumi@1
  displayName: 'Pulumi up'
  inputs:
    azureSubscription: '$(azureSubscription)'
    command: up
    args: '--yes'
    cwd: '$(workingDir)'
    stack: '$(pulumiStack)'
    createStack: true

Is there a better way to do this or can I request one?

[praneetloke]

@darnmason pulumi not installing the plugins automatically when you use the binary option is expected since the Pulumi CLI doesn't support it when you use the binary option. Another related issue is https://github.com/pulumi/pulumi/issues/5334 (and specifically this comment). cc @mikhailshilkov just in case he is aware of anymore options for .NET binaries specifically. My ideal solution would be for the CLI would be to emit a manifest file of sorts for language runtimes that support pre-built binaries, so that users could use that manifest to tell the CLI to install plugins from that with a single command, or better yet detect the presence of the manifest file in the current folder and automatically install from that.

The current workaround that I myself use sometimes is to add a CmdLine task step after the Pulumi CLI has been installed by a previous Pulumi task step. (I have long wanted to allow adding the Pulumi task without requiring a command to be selected so that the task would simply install the CLI and bail because no command was provided. Today you have to specify a command.) Once the CLI is install, I then add all the pulumi plugin install ... commands in the single CmdLine task step as a multi-line script statement. It's not ideal but it is better than having to repeat the Pulumi task step for every plugin you need to install.

And for a better way to install the plugins for pre-built binaries, would it be helpful if the task supported a special "custom" command, where you could simply enter all the pulumi plugin install ... commands into the task itself and let it run those after installing the CLI? I would love to hear if you have any other thoughts on this as well.

[praneetloke]

@darnmason since v1.0.9 you should be able to use the task extension to simply install the CLI and then run a single script task to install all plugins. It's still not the best experience but slightly better than having to specify the Pulumi task multiple times for each plugin you want to install.

- task: Pulumi@1
  displayName: Install Pulumi CLI
- script: |
    pulumi plugin install resource azure-native v0.7.1
    pulumi plugin install resource random v3.1.0
    ...
    ...
- task: Pulumi@1
  input:
    cmd: up
    args: --yes
    stack: mystack

(Note to other readers: you are only need to install the plugins manually if you are using a pre-built binary with Pulumi at this time.)

[darnmason]

Definitely an improvement in the usability of the Pulumi task, using it just to install the tools.

Still a bit to go with automatically installing the plugins. A pain point is having the hardcoded plugin versions in your deployment script. You update a plugin in Nuget and forget to update the deployment script and need to fix and redeploy. Removing this manual step should be the end goal.

If it were possible for Pulumi to write a manifest file during a regular dotnet build/publish and automatically detect that manifest when running from binary that'd be great. The onus would be on the developer to ensure that manifest is copied when moving files which makes sense.

[ModernRonin]

it would be extremely helpful if the command pulumi plugin install supported "latest" for version.

use-case: I have a nightly pipeline running a docker-image to use by our pulumi-related pipelines. that image is based on pulumi/pulumi-nodejs because we are using TS. however, we use several plugins.

now I could just let pulumi install them whenever it's invoked, but that's wasting time - the whole purpose of the nightly built pulumi image is enabling the pipelines to use pulumi as fast as possible.

so I put the pulumi plugin install lines in the dockerfile - however, because I have to specify a version number, this will pin it to that version instead of getting the latest one.

[lukehoban]

it would be extremely helpful if the command pulumi plugin install supported "latest" for version.

Agreed that this would be a useful addition. I've opened https://github.com/pulumi/pulumi/issues/9001 to track that directly.

[rdanno]

Not sure if this is the thread for it. I noticed that the prior version of a plugin needs to exist when performing a deploy after installing a new plugin. The state references the version of the initial plugin. Only after the deploy is performed can the original plugin be removed. It's only a problem if the first plugin is removed before the new one is installed. It also means the plugin cache tends to fill up with old versions which might not be needed.

[lukehoban]

@rdanno I expect what you are describing is due to https://github.com/pulumi/pulumi/issues/5619.