Clarify third party plugins distribution model and usage of builder / `yarn plugin import`

[abdes]

Plugins are one of the best features introduced to berry. It remains however a bit unclear in the current documentation and in the current behavior of yarnpkg-builder how this feature fits into the larger ecosystem (i.e. outside the internal use of plugins by berry itself).

• builder build plugin always normalizes the plugin name to '@yarnpkg/xxxxx'. It's not clear today if this is intended or if there is something more that we need to know,
• the tutorial for plugin development does not use the yarn plugin command line, which behaves slightly differently from the tutorial example. Here again we may need a bit more clarity on the best practice of how plugins get added to a berry project,
• yarn plugin import has different treatment depending on whether the plugin is hosted on yarnpkg repo or not. If not, it has to be a downloadable single file bundle. Why we can't just ship third-party plugins as npm modules, add them as dev deps, and then yarn can just import from there, like usual?
• even when a plugin is imported from the local filesystem, yarn plugin import builds a different cjs file. Not clear how is that working with builder and if we (as plugin developers) need to know about it.

[arcanis]

It remains however a bit unclear in the current documentation and in the current behavior of yarnpkg-builder how this feature fits into the larger ecosystem

That's true! It's because we didn't put much thought into that yet 😅

Given the already large number of changes in the v2, we preferred to implement low-level tools and figure out the next high-level layers based on community feedback during the next months. Plugins are one of those things that are a bit rough at the moment until we can figure out a better user story.

builder build plugin always normalizes the plugin name to '@yarnpkg/xxxxx'. It's not clear today if this is intended or if there is something more that we need to know,

It's intended...ish. The normalization was made so that people could in theory override the builtin plugins (like essentials) with their own implementations. In practice I'm not sure it works, and I'm even less sure it's a valid use case. I'd be ok with removing this limitation.

yarn plugin import has different treatment depending on whether the plugin is hosted on yarnpkg repo or not. If not, it has to be a downloadable single file bundle. Why we can't just ship third-party plugins as npm modules, add them as dev deps, and then yarn can just import from there, like usual?

Because, just like the Yarn bundle itself, plugins cannot have dependencies (chicken and egg problem; if you have dependencies you need to run an install, but to run an install you need the plugins to be registered).

So I figured the easiest way to distribute them would be as single JS files (just like the regular CLI), and thus using the npm registry didn't really provide anything of value (since you need to download a tgz which may contain many files).

even when a plugin is imported from the local filesystem, yarn plugin import builds a different cjs file. Not clear how is that working with builder and if we (as plugin developers) need to know about it.

Not sure what you mean - iirc we just copy it into the local directory and add it to the configuration.

[abdes]

Thank you @arcanis for such a quick turnaround!

even when a plugin is imported from the local filesystem, yarn plugin import builds a different cjs file. Not clear how is that working with builder and if we (as plugin developers) need to know about it.

Not sure what you mean - iirc we just copy it into the local directory and add it to the configuration.

Actually you are right, only the bundle file name is changed from .js to .cjs. Not sure again if it is intended or not that builder produces .js and plugin import makes it .cjs.

I'm starting a repo in which I will be experimenting with writing a plugin to analyze and visualize the dependency graph inside a multi-workspace berry project. It will be taking the output from workspaces list as json and then starting a local web server to serve an app that will visualize and offer analysis tools for that graph. I chose a complex scenario involved more than the manifests and the core API in order to push the plugin model to the edge :-). I will share my findings and challenges as I make progress on it.

The initial thoughts on the structure are as follows:

• project will deliver a single package containing the app and the plugin, that package can have any scope/name, not necessarily @yarnpkg so it can be published independently to the npm registry,
• the package is a dev dependency of the root workspace and gets installed by yarn as usual,
• the package is unplugged and as postinstall, it will import its plugin into yarn from the local filesystem, Not sure if that will work, but I will try :-)
• the package is also unplugged because the plugin will need access to the static files inside the app
• plugin is bundled in a single file with builder, still with the @yarnpkg scope (forced by builder), although I do agree with you that it is still not clear why is that happening if it's a third party, not a core plugin :-). So I'm not gonna webpack the plugin myself, just let builder do whatever it does.
• the plugin will not return control to yarn until the user stops the command (Ctrl-C), will see if that's an issue.

Let me know if something seems wrong to you in the approach described above.

Again, really appreciate the work done on berry and your feedback.

[arcanis]

• I'm not sure why you want to have both a plugin and a package - I'd suggest to avoid that and simply include everything into the plugin. If you want to have a package to use it with other package managers too, imo you should just bundle it inside the plugin (the builder will do this) and just distribute the Yarn plugin to users.

• I'd also avoid the postinstall and the unplug. I don't know how many statics you have, but with the current single-file model you'd rather bundle the assets inside the JS - it's a bit weird, I agree, but usually there aren't that many images within plugins 😄

[abdes]

• I'm not sure why you want to have both a plugin and a package - I'd suggest to avoid that and simply include everything into the plugin. If you want to have a package to use it with other package managers too, imo you should just bundle it inside the plugin (the builder will do this) and just distribute the Yarn plugin to users.

The first issue is how to distribute the plugin to the users. People just do git clone and yarn install and expect everything to be ready. Now we will expect them to do another step as well: yarn plugin import https://xxxxx/plugin.js.

• I'd also avoid the postinstall and the unplug. I don't know how many statics you have, but with the current single-file model you'd rather bundle the assets inside the JS - it's a bit weird, I agree, but usually there aren't that many images within plugins 😄

• postinstall is to handle the yarn plugin import on behalf of the user. so we just stay with yarn install
• builder cannot handle bundling of a react app with style loaders, a UI components library and other stuff... anyway it's not supposed to handle that. I'm pushing the boundary here of what a plugin could do, but the whole point is to test the toolchain for a complex plugin.

The app packing can be done outside in a separate workspace though. We can bundle everything inside a webpack library single chunk and pass it as a dependency to the plugin so builder can still produce a single file. The impacts of that approach is that I need to find a way to serve the app script to the browser out of the plugin bundle :-). Not sure how...

I will play around this weekend with both approaches and let you know after.

[arcanis]

The first issue is how to distribute the plugin to the users. People just do git clone and yarn install and expect everything to be ready. Now we will expect them to do another step as well: yarn plugin import https://xxxxx/plugin.js.

Note that plugins are meant to be checked-in (same as the Yarn binary itself), so cloning is all that's required to have a working environment.

[abdes]

Can a plugin access the files inside a package in the cache?

I tried the example at: https://yarnpkg.com/features/pnp#frequently-asked-questions

const {readFileSync} = require(`fs`);

// Looks similar to `/path/to/.yarn/cache/lodash-npm-4.17.11-1c592398b2-8b49646c65.zip/node_modules/lodash/ceil.js`
const lodashCeilPath = require.resolve(`lodash/ceil`);

console.log(readFileSync(lodashCeilPath));

I got the following error:

TypeError: Cannot read property '1' of undefined
    at isFileType (fs.js:177:16)
    at Object.readFileSync (fs.js:363:16)
    at l._serveStaticContent (E:\dev\projects\workspaces\yarn-plugins\.yarn\plugins\@yarnpkg\plugin-workspaces-analyze.cjs:5:890)
    at Server.<anonymous> (E:\dev\projects\workspaces\yarn-plugins\.yarn\plugins\@yarnpkg\plugin-workspaces-analyze.cjs:5:1277)    at Server.emit (events.js:315:20)
    at parserOnIncoming (_http_server.js:790:12)
    at HTTPParser.parserOnHeadersComplete (_http_common.js:119:17)

There are a number of issues related with that access model anyway:

1. In code bundled with webpack, the require will be transformed by webpack. I tried many ways to get that to be bypassed with no success: eval('require'), eval('require.resolve'), __non_webpack_require__ but undefined, ...
2. What about accessing and loading non .js files?

Is there another way to acess the zip file for a package in the cache other then using require?

[abdes]

After more tests, I came to the conclusion that because plugins cannot have dependencies and pretty much won't work unless they are really limited to a single file bundle, the most appropriate use cases for them is to execute commands that only use @yarnpkg modules. Any more complicated operations that require external dependencies are better done in separate cli from yarn.

The node fs implementation when running the cli with yarn node works fine with yarn PnP.

[abdes]

After more tests, I came to the conclusion that because plugins cannot have dependencies and pretty much won't work unless they are really limited to a single file bundle, the most appropriate use cases for them is to execute commands that only use @yarnpkg modules. Any more complicated operations that require external dependencies are better done in separate cli from yarn.

The node fs implementation when running the cli with yarn node works fine with yarn PnP.

So, yeah much simpler to keep complicated stuff out of yarn plugins and just reuse as much as possible the existing plugins. So far I've not encountered the need yet to build an extra plugin :-)

You can have a look at the web UI for project dependencies simply by using yarn dlx from any project root:

yarn dlx -p @lerepo/cli lrt web

The cli brings the web app as a dependency and serves static files seamlessly through node fs provided by yarn. Works 100% with PnP, no unplugging needed.

[ghost]

@arcanis is the intention for yarn plugins only to be those that strictly interact with the @yarnpkg/* ecosystem and a select set of modules exposed by the require function that factory exposes?

I am trying to assess whether yarn 2 is going to be a good fit for projects or if we should just migrate to npm. The differentiating factor for us is plugins, as the other features are either already there in npm (especially now that it supports workspaces) or are quickly being developed, but no other package manager has this concept of plugins. I like that they're fat 'binaries' if you will, but not being able to cleanly package outside modules is a real problem. We want to leverage this functionality to have a set of core plugins that take over various things (like prettier, eslint etc) so we can just share these plugins between projects instead of sharing a config + ensuring the correct dependencies are installed (and importantly versions). Right now we have to manage versions and consistency and its really easy for things to fall out of date, where is shipping a plugin and updating that plugging infrastructure wise, would be really easy (distribution isn't the problem we have, its time and effort). For us, not having to run updates against the package.json, but instead just importing a plugin, is something that fits really well with the mental model.

I think its yarns killer feature now, frankly, and I'd like to know more about this because it would be really great to package up these kind of tools as part of the package manager rather than building separate tools to manage it all.

[arcanis]

is the intention for yarn plugins only to be those that strictly interact with the @yarnpkg/* ecosystem and a select set of modules exposed by the require function that factory exposes?

Yes and no - plugins can't access third-party packages out of the box because it would mean that they wouldn't be able to execute until after the project has been installed (and thus after the plugins have already been executed).

That being said, you can bundle your plugin sources into a single binary (like we do for Yarn), in which case you can use whatever third-party you want since they'll be stored inside the plugin anyway. This is what @yarnpkg/builder does for you, although it's not well enough documented at the moment.