Provider update: resource plugin aws is expected to have version

SteffenBoThomsen commented 1 year ago

What happened?

I've recently upgraded the provider plugin @pulumi/aws to a newer version. Now when i do a preview or up I get the following warning:

warning: resource plugin aws is expected to have version >=5.27.0, but has 5.9.2; the wrong version may be on your path, or this may be a bug in the plugin

When doing a pulumi plugin ls I see the following:

NAME          KIND      VERSION  SIZE    INSTALLED       LAST USED
aws           resource  5.27.0   499 MB  1 hour ago      1 hour ago
aws           resource  5.9.2    387 MB  48 minutes ago  48 minutes ago

If I run the following command to remove the 5.9.2 plugin version pulumi plugin rm resource aws 5.9.2 and then run a preview I get the following error:

error: could not load plugin for aws provider 'urn:pulumi:test::sapbasis::pulumi:providers:aws::default_5_27_0': aws (resource) plugin [C:\Program Files\chocolatey\bin\p
ulumi-resource-aws.exe] wrote a non-numeric port to stdout ('0'): strconv.Atoi: parsing "Cannot find file at ..\\\\lib\\pulumi\\home\\plugins\\resource-aws-v5.9.2\\pulu
mi-resource-aws.exe' This usually indicates a missing or moved file.\r": invalid syntax

This indicates that the stack is somehow still using the 5.9.2 Provider. Doing an export of the stack with pulumi stack export --file out.json - the stack output has no references to the 5.9.2 version but are instead all listed as using ::pulumi:providers:aws::default_5_27_0.

This lets me believe that either:

The warning in itself is the bug, or
The stack is actually using the older provider plugin, but setting the new provider version on the resources on pulumi up

Let me know if I can provide further information that can help track this error.

Steps to reproduce

I haven't been able to reproduce the setup.

Expected Behavior

I expected to do the update of the provider plugin, and then see that the provider would be updated on resources on the next pulumi up without giving me warnings on the provider version.

I wouldn't expect the program to give me an error after removing the old plugin version, when the stack is pointing to an available newer version.

Actual Behavior

I get a warning on a provider version no longer in use - and can't remove the old provider plugin as the stack then fails.

Output of `pulumi about`

No response

Additional context

An observation, that might be a clue, our setup is a mono-repository where we have multiple stacks. All got the provider updated, but only some of the stack are printing the warning. The common denominator between the stacks that print the warning are, they all use custom resource components.

Contributing

Vote on this issue by adding a 👍 reaction. To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

dixler commented 1 year ago

Hi. Thanks for filing this. I am having difficulty reproducing this with the information given. I have a few questions

In terms of a workaround, does downgrading @pulumi/aws back to the previous version fix the issue?
Can you post the output of pulumi about?
What package manager do you use with your monorepo?

The common denominator between the stacks that print the warning are, they all use custom resource components.

This seems interesting. I'm trying to drill down on this. Can you elaborate more on what custom resource components you're using? For example, we have ComponentResource and CustomResource.

Thanks!

SteffenBoThomsen commented 1 year ago

Hi,

In terms of a workaround, does downgrading @pulumi/aws back to the previous version fix the issue?

After downgrading I still get the error, but here it seems valid enough, as the stacks resources are set to use the provider 5.27.0 and it now actually is back at using 5.9.2.

Can you post the output of pulumi about?

Sure, though there's no options to tell pulumi where to look for the yarn.lock file, in a monorepo where we utilize NX - the stacks do not reside together with the lock file.

pulumi about

```text CLI Version 3.51.1 Go Version go1.19.4 Go Compiler gc Plugins NAME VERSION nodejs unknown Host OS Microsoft Windows 10 Enterprise Version 10.0.19045 Build 19045 Arch x86_64 This project is written in nodejs: executable='C:\Program Files\nodejs\node.exe' version='v18.6.0' Current Stack: {ORG}/sapbasis/test TYPE URN pulumi:pulumi:Stack urn:pulumi:test::sapbasis::pulumi:pulumi:Stack::sapbasis-test pulumi:providers:aws urn:pulumi:test::sapbasis::pulumi:providers:aws::default_5_27_0 sappi:aws:SsmDistributor urn:pulumi:test::sapbasis::sappi:aws:SsmDistributor::test-ssm-distributor-eu-west-1 aws:kms/key:Key urn:pulumi:test::sapbasis::sappi:aws:SsmDistributor$aws:kms/key:Key::test-ssm-distributor-assets-eu-west-1-kms-cmk aws:kms/alias:Alias urn:pulumi:test::sapbasis::sappi:aws:SsmDistributor$aws:kms/alias:Alias::test-ssm-distributor-assets-eu-west-1-kms-cmk-alias aws:s3/bucket:Bucket urn:pulumi:test::sapbasis::sappi:aws:SsmDistributor$aws:s3/bucket:Bucket::test-ssm-distributor-assets-eu-west-1-s3 aws:ssm/parameter:Parameter urn:pulumi:test::sapbasis::sappi:aws:SsmDistributor$aws:ssm/parameter:Parameter::test-ssm-distributor-elastic-cloud-id-eu-west-1-ssm-p arameter aws:ssm/parameter:Parameter urn:pulumi:test::sapbasis::sappi:aws:SsmDistributor$aws:ssm/parameter:Parameter::test-ssm-distributor-elastic-cloud-auth-eu-west-1-ssm -parameter sappi:aws:SsmFilebeatDistribution urn:pulumi:test::sapbasis::sappi:aws:SsmFilebeatDistribution::test-filebeat-distribution-eu-west-1 aws:s3/bucketObject:BucketObject urn:pulumi:test::sapbasis::sappi:aws:SsmFilebeatDistribution$aws:s3/bucketObject:BucketObject::test-filebeat-package-suse-installer-eu -west-1-s3-object aws:ssm/document:Document urn:pulumi:test::sapbasis::sappi:aws:SsmFilebeatDistribution$aws:ssm/document:Document::test-filebeat-package-eu-west-1-ssm-document Found no pending operations associated with test Backend Name pulumi.com URL https://app.pulumi.com/steffenthomsen User steffenthomsen Organizations steffenthomsen, {ORG} warning: Failed to get information about the Pulumi program's dependencies: could not find either yarn.lock or package-lock.json ```

What package manager do you use with your monorepo?

Yarn

This seems interesting. I'm trying to drill down on this. Can you elaborate more on what custom resource components you're using? For example, we have ComponentResource and CustomResource.

We are using CustomResource components, but this morning, the observation was disproved. The only other thing I can think of, is that the stacks that give the warning, have been previewed (directly after upgrade) using the Automation API. Whereas the stacks that have been previewed with the cli did not have the message. I'm not 100% that this is true throughout - but it's the only link I can think of.

EDIT: Just ran a test on whether the automation-api could be the culprit, and it doesn't seem like it as I get the warning on a completely new stack both with previews through cli only and through the auomation-api.

Frassle commented 1 year ago

Ah! This took longer than I'd like to admit to see, but

ulumi-resource-aws.exe] wrote a non-numeric port to stdout ('0'): strconv.Atoi: parsing "Cannot find file at ..\\\\lib\\pulumi\\home\\plugins\\resource-aws-v5.9.2\\pulu
mi-resource-aws.exe' This usually indicates a missing or moved file.\r": invalid syntax

That error message gives a big hint to what I think has gone wrong. You've got an executable called "pulumi-resource-aws.exe" on your PATH (or Path, because Windows). Pulumi always prefers to use executables on PATH over those in the plugin directory (this should probably be changed, but lord knows how many people now depend on this behavior).

I'm not sure how you've ended up with a resource provider in C:\Program Files\chocolatey\bin but I'd suggest you delete it, then try again.

SteffenBoThomsen commented 1 year ago

I'm not sure how you've ended up with a resource provider in C:\Program Files\chocolatey\bin but I'd suggest you delete it, then try again.

Removing them also removes the warning, so that's perfect.

The following list is the resource providers present in the bin path:

pulumi-resource-docker.exe
pulumi-resource-command.exe
pulumi-resource-azure-native.exe
pulumi-resource-awsx.exe
pulumi-resource-aws.exe

Azure Native is a quite recent addition, as it's only a couple of days ago i created the first azure project. I used pulumi new azure-typescript - i think the solution might be to ignore resource provider executables in the chocolatey package installation:

https://docs.chocolatey.org/en-us/create/functions/install-binfile

Chocolatey automatically shims executables in package folders that are not explicitly ignored, putting them into the bin folder (and subsequently onto the PATH).

Frassle commented 1 year ago

i think the solution might be to ignore resource provider executables in the chocolatey package installation:

They already should be. Provider executable aren't even fetched via chocolatey. I wonder if you've got an unusual PULUMI_HOME environment variable set that is causing these to get installed to chocolateys folder rather than the user home folder?

SteffenBoThomsen commented 1 year ago

They already should be. Provider executable aren't even fetched via chocolatey. I wonder if you've got an unusual PULUMI_HOME environment variable set that is causing these to get installed to chocolateys folder rather than the user home folder?

Spot on.

SteffenBoThomsen commented 1 year ago

I wonder if you've got an unusual PULUMI_HOME environment variable set that is causing these to get installed to chocolateys folder rather than the user home folder?

To expand on this part. I was indeed the case. Due to tight execution policy on corporate machines, it could not be located in the default location. So when initially setting up Pulumi it was changed. For ease of file structure I placed it together with the actual installation.

This resulted in the above problem. As chocolatey would indeed take all executables in the chocolatey tree and put them into the bin folder if they where not ignored. Probably a very niche case. Only thing I can think of, that might have avoided it in my case, would be a note on the docs around PULUMI_HOME.

Frassle commented 1 year ago

Thank you so much for confirming!

Only thing I can think of, that might have avoided it in my case, would be a note on the docs around PULUMI_HOME.

I've raised https://github.com/pulumi/pulumi-chocolatey/issues/11 to track that.

pulumi / pulumi