yarn pnp support

[brandonkal]

On my machine, a kubernetes-typescript install takes: 200 seconds with a standard install 20 seconds with yarn PNP

First-class Yarn PNP support would be appreciated as it would mean much faster project startup time.

Error:

brandon@dev:~/pulumi-project/01-first$ pulumi up
Enter your passphrase to unlock config/secrets
    (set PULUMI_CONFIG_PASSPHRASE to remember): 
Previewing update (test):
     Type                 Name                 P
     pulumi:pulumi:Stack  pulumi-project-test   

Diagnostics:
  pulumi:pulumi:Stack (pulumi-project-test):
    error: It looks like the Pulumi SDK has not been installed. Have you run npm install or yarn install?

error: failed to load language plugin nodejs: could not read plugin [/home/brandon/.pulumi/bin/pulumi-language-nodejs] stdout: EOF

Reproduction

pulumi new kubernetes-typescript
mkdir 01-first && mv package.json 01-first/
mv index.ts 01-first/
echo 'config: .pulumi' >> Pulumi.yaml

Create a package.json

{
  "private": true,
  "workspaces": [
    "01-first"
  ]
}

yarn policies set-version v2
yarn install
cd 01-first
pulumi stack init test

The tree should now look like this: tree

.
|____yarn.lock
|____tsconfig.json
|____.yarn
  ...snip...
|____.gitignore
|____.pnp.js
|____01-first
| |____index.ts
| |____package.json
|____.pulumi
| |____Pulumi.test.yaml
|____Pulumi.yaml
|____package.json
|____.yarnrc

pulumi up
// logs error noted above

[pgavlin]

Interesting! This certainly sounds like a promising improvement.

[ringods]

A first stable version of Yarn 2 has been announced on 24 January 2020:

https://dev.to/arcanis/introducing-yarn-2-4eh1

One of the biggest changes is that Plug'n'Play is enabled by default. You can expect that more and more people will start using it. I just tried it too, only to see it failing and eventually find this GH issue.

Looking at the Migration Guide, I think you do not need to support PnP specially. If Pulumi can detect a yarn enabled project (yarn.lock right?), it should launch yarn node rather than just node. I assume yarn will then detect whether the project has PnP enabled or not and calculate the correct NODE_PATH.

[ringods]

Looking at the NodeJS shim, I see some functionality assuming there is a node_modules folder.

https://github.com/pulumi/pulumi/blob/930adc05047dfe5c3412799fa05c00cd6acd0183/sdk/nodejs/cmd/pulumi-language-nodejs/main.go

This will make it a bit harder to get Yarn 2 support in I guess. 😕

[ringods]

For the people who already want to use the other nice features of Yarn 2, but don't be locked out due to PnP, you can still disable it. I found how to disable it in this StackOverflow answer:

.yarnrc.yml:

nodeLinker: node-modules

[amurdock]

Just in case this helps anyone out and I appreciate it's only a stop-gap however yarn add @yarnpkg/pnpify -D and running pnpify pulumi up works for me 👍🏻

[RobbieMcKinstry]

Would adding an official template here help? When you create a project, you would run...

$ pulumi new typescript-pnp

...to get a project skeleton using Yarn Berry PNP instead of NPM.

Supposing this was possible, WDYT? cc: @ringods and anyone who wants to chime in! :^D

[ringods]

@RobbieMcKinstry Yarn Berry PNP doesn't work at the moment. Some time ago, I tried to submit a PR but failed because there are so many places within the Pulumi CLI which expects a node_modules folder to be there: #4349

[RobbieMcKinstry]

Dang... I was hopeful there might be an expedient solution.

[RobbieMcKinstry]

Long term, I want us to decouple the SDKs from ecosystem tools. For example, Python couples venv tightly, making it harder to use Poetry. Another example is TS-Node; it would useful to allow a different transpiler, like SWC or ESBuild, to be plugged in instead. We have open issues to track these tools individually, but I want to propose an epic to propose an approach to tackle the unifying theme.

[NexZhu]

Reporting the same problem for Pulumi project in a pnpm workspace, strangely it used to work before..

[RobbieMcKinstry]

Note to my fellow Pulumi maintainers: anyone coming back to this issue should start reading here. @ringods did a great job summarizing the state of the world and the core challenges faced in implementing pnp support.