Yarn 3 + Framework Tools: Potential next-step improvements

[thedavidprice]

cc @peterp

Continues the work from PR #3154. For context:

• https://github.com/redwoodjs/redwood/pull/3154#issue-957115489
• https://github.com/redwoodjs/redwood/pull/3154#issuecomment-922070030

Yarn 3 "scopes" binaries much more strictly than Yarn 1. To access a bin, it must be a dependency in the workspace directory or in the root package.json. This works a bit against the original purpose of our tasks/framework-tools directory, which provided a way to separate concerns between package dependencies and monorepo tooling + scripts. Because framework-tools is neither a workspace nor in root, the scripts wouldn't run bins unless they were dependencies in the workspaces. Although there are now Yarn utilities like constraints, interactive, deduce, and 'workspaces foreach' to better manage things, I had to install many of the dependencies from framework-tools across all the workspaces.

Decisions ahead:

1. we can leave things as is: same structure with redundancy of dependencies across workspaces
2. or we can re-structure our tooling scripts (most likely move the tasks/framework-tools back into root)

I'm ok with 1. I think we should seriously consider option 2:

• replace Lerna with workspaces foreach (publishing needs special consideration)
• reduce script redundancy (and leverage Yarn's new sharing scripts)
• remove all the dependencies added across packages

It simplifies things, which is what I like most about it.

References for next steps

Yarn: sharing scripts

Will require re-architecting scripts:

• https://yarnpkg.com/getting-started/qa/#how-to-share-scripts-between-workspaces

Using workspaces foreach

Replace lerna run for build, lint, test, etc.

• https://yarnpkg.com/cli/workspaces/foreach

Replace Lerna publishing

This needs more evaluation to weigh pros/cons. Resources and examples:

Monodeploy Designed specifically for use in CI (does not auto-commit during publishing):

• https://github.com/tophat/monodeploy

  Monodeploy is a powerful tool which aims to simplify the package publishing process for monorepos. It leverages Yarn Berry workspaces to do the heavy lifting, and is a direct replacement for tools such as Lerna and Semantic Release.

Roll our own Babel plugin See Babel project example:

• https://github.com/babel/babel/pull/12138
• https://github.com/nicolo-ribaudo/yarn-plugin-babel-release-tool

Misc

Also of interest but needs further research.

Yarn build plugin https://yarn.build/

BUILD and BUNDLE your local workspaces. It's like Bazel, Buck and Pants but for Yarn

[ojkelly]

👋 Author of yarn.build here, if you need any help deciding if it's of use or the right tool I'm happy to answer any questions.

I've had a bit of a look around (neat framework!), I think you'll find value in the yarn build command, but might want something like monodeploy for publishing to npm.

yarn build will build a single workspace or all of them I parallel, utilising a cache to ensure only what needs to be built is. It builds in the correct order according to your dependencies.

yarn bundle is designed to bundle a nodejs app from a workspace in your monorepo. It creates a zip file, or outputs to a directory, that contains just the workspace and its dependencies. It's designed for building artefacts to deploy to Lambda, Docker, etc.

yarn test similar to the build command, except it will run all the tests across all workspaces in your project.

You can also open a discussion on our repo if this ticket isn't the best place https://github.com/ojkelly/yarn.build/discussions/new?category=q-a

[jtoar]

Hey @ojkelly, thanks for reaching out! We only recently made the switch to yarn 3 so we've still yet to really take advantage of its features. It's helped us out tremendously already though.

I gave yarn build a spin in #3851 and it seems promising so far! I need to put together a few more questions, but one I had right away was around watching for file changes. Does yarn.build have a recommended way of doing that?

I looked around at a few other monorepos and they all seem to build in a different way. Jest uses a script, yarn itself uses a root command I think (i.e. taking advantage of the fact that scripts named with colons are globally available). Lastly I checked out yarn.build which seems closer to what we're doing (each package has a build script). I'm sure each has pros and cons, but I was wondering if you had any best practices in mind around building a monorepo?

[ojkelly]

File watching is a surprisingly tricky and hard to scale problem.

Notably https://github.com/facebook/watchman is probably on of the more advanced solutions, and one of the better ones in a monorepo where multiple applications might want to subscribe to change notifications.

My approach is again to keep it simple.

Each workspace should own its own development loop. Yarn.build comes in when you want to ensure consistency across workspaces in a repo.

build and test should be callable from outside the workspace so you can co-ordinate cross-cutting changes, and validate your work hasn't broken anything else.

But watching is part of the devloop, and to me makes sense on one or a couple or workspaces at most. You could watch all files from the root and re-run yarn.build's build if you wanted to. Just checked your PR and saw you did just that. 👍

Jest running from the top level, seems to always be a bit weird.

I don't think there's a hard and fast rule, as it really depends on the tooling. But it has to be fast and reliable.

To me, Monorepo tooling really just needs to get out of the way, if it isn't you're going to spend more time on it than features.

[orta]

I did a quick writeup a while back on building watcher for the typescript website monorepo with watchman, it's pretty simple to do once you start working on it: https://github.com/microsoft/TypeScript-Website/issues/130#issuecomment-664673740