Private Providers Hosted in Pulumi Cloud

[MMartyn]

Hello!

I am currently building internal MLC providers for my company and there is a bit of friction in getting those providers downloaded where they are needed, such as the deployment agents. It would be very beneficial if there were a mechanism built into Pulumi Cloud that would allow me to host those for the organization that the agents and cli would automatically understand how to download. I am thinking something along the lines of how Terraform Cloud allows you to have private modules.

• Vote on this issue by adding a 👍 reaction
• If you want to implement this feature, comment to let us know (we'll work with you on design, scheduling, etc.)

Issue details

Affected area/feature

Pulumi Service

[justinvp]

there is a bit of friction in getting those providers downloaded where they are needed, such as the deployment agents

Can you provide more details on the friction?

[ringods]

@MMartyn when building your providers, you can use the pluginDownloadUrl to point to a private location:

Here is an example from a non-Pulumi owned package, living in another Github organization: https://github.com/pulumiverse/pulumi-unifi/blob/main/provider/cmd/pulumi-resource-unifi/schema.json#L14

You can have private Github releases for the plugin binaries because Pulumi picks up the GITHUB_TOKEN to authenticate. Your SDKs can be published to a private registry like Artifactory or Nexus. Hope this already helps a bit.

[MMartyn]

there is a bit of friction in getting those providers downloaded where they are needed, such as the deployment agents

Can you provide more details on the friction?

Some of the friction is needing to have the consumers set up their GITHUB_TOKEN in order to pull the binaries and another would be needing to specify the providers explicitly in the stacks like so:

provider:
    type: pulumi:providers:foo
    defaultProvider: true
    options:
      version: "0.0.1"
      pluginDownloadURL: github://api.github.com/org/foo

[MMartyn]

You can have private Github releases for the plugin binaries because Pulumi picks up the GITHUB_TOKEN to authenticate

Once, I set the download url on the provider, I see that the deployment agent was able to download the provider, so seems deployments are ok in this regard.

[ringods]

@MMartyn the need to set the pluginDownloadUrl when using Pulumi YAML is needed for any non-Pulumi owned package, public and private.

https://www.pulumi.com/docs/languages-sdks/yaml/yaml-language-reference/#resource-options

This is not needed for regular language SDKs as this url is integrated in the generated SDKs, for instance:

https://github.com/pulumiverse/pulumi-unifi/blob/main/sdk/nodejs/package.json#L27

[MMartyn]

The main pain point that pushed me to opening this was that it at first seemed like I would need to manually setup a GITHUB_TOKEN in the deployment agent for private providers to work. However, it ended up not being the case as the agent already had a token it could use to get the private provider (although it isn't clear to me how that mechanism works). So that said, the only remaining parts that I find a bit of a chore are ensuring users set up their GITHUB_TOKEN's and finding an appropriate place to host the SDKs.

Feel free to close this as my main issue ended up being fine.

[raysango]

I'm also using a private Github repo to host my provider, When I set the pluginDownloadURL to the artificat download url and set GITHUB_TOKEN I get a 404 HTTP error fetching plugin from https://github.com/{ORG}/{REPO}/releases/download/v1.3.1/pulumi-resource-vital-v1.3.1-darwin-amd64.tar.gz

[ringods]

@raysango did you leverage the explicit Github release support?

https://www.pulumi.com/docs/iac/packages-and-automation/pulumi-packages/authoring/#support-for-github-releases

Can you verify that you can download the file using curl and a Personal Access Token?

https://stackoverflow.com/a/35688093