Workspaces don't run lifecycle scripts when linking local packages

[jquense]

What is the current behavior?

When you do a yarn install in a workspaces enabled project, yarn doesn't run the various lifecycle scripts for workspace packages, e.g. prepare prepublish preinstall etc

If the current behavior is a bug, please provide the steps to reproduce.

Here is a commit adding a failing test case: https://github.com/jquense/yarn/commit/61eaf4239f707f772af5713ebc98963589a6dad4

What is the expected behavior?

Lifecycle scripts are run for the local packages so they have an opportunity to build steps. This is what lerna does.

Please mention your node.js, yarn and operating system version.

yarn 27.5 node 8 Mac Sierra

cc @bestander

[bestander]

PR is welcome BTW :)

[jquense]

I took a stab at it, but could figure out the code base unfortunately. All I can see is that scripts is empty for the linked packages when it gets to the getInstallCommands in the pacakge-install-scripts :/

[bestander]

Thanks for giving it a try, @jquense. The entry point is probably here https://github.com/yarnpkg/yarn/blob/master/src/cli/commands/install.js#L477 - flattenedTopLevelPatterns only contains the top level dependencies of root package.json + a virtual dependency that refers all workspaces. I think going deeper PackageInstallScripts should handle that https://github.com/yarnpkg/yarn/blob/master/src/package-install-scripts.js#L248.

If you want you can start with submitting a PR with a failing test and we can sort it out in a follow up.

[jquense]

The failing test case linked in the issue is the best I can do sorry! Unfortunately I just don't have the time to spend hours learning the yarn codebase to try and fix this.

[BurtHarris]

Hi. I'm quite interested in yarn and workspaces in particular. I'll be giving a look at this as a possible first contribution.

[bestander]

Awesome! If you have any questions ask away

On Mon, Jul 31, 2017 at 7:28 PM Burt Harris notifications@github.com wrote:

Hi. I'm quite interested in yarn and workspaces in particular. I'll be giving a look at this as a possible first contribution.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/yarnpkg/yarn/issues/3911#issuecomment-319250162, or mute the thread https://github.com/notifications/unsubscribe-auth/ACBdWBmqV2fFlBlOvOlXB77Ml0213Xw6ks5sTo1UgaJpZM4OUyrV .

[BurtHarris]

@bestander I hit a snag right away, Issue #4059. Its almost certainly related to the fact I'm running Windows any hints on how to make that work?

[bestander]

As a workaround you should be able to run tests like this node ./node_modules/jest/bin/jest.js

[jquense]

Any word from anyone? I might have some time to jump, and I really would love to get this fixed :) I have to think that for most folks the lack of lifecycle scripts running is a dealbreaker for this feature?

[bestander]

The core team is focused on fixing the most high priority issues right now, cases where Yarn may crash or install wrong versions of dependencies.

This bug will be resolved eventually but considering that this is a community project the quickest way would be to spend time learning the codebase and sending a fix.

[jquense]

Definately :) I'm not trying to be pushy or entitled just wanted to get a sense of where the issue was and if there was any existing work

[ryanhiebert]

FWIW, it's not the blocker for my adoption of workspaces. I haven't figured out how to reliable add and remove dependencies (not unlikely to be user error). Handling this was annoying, but I just needed to add calls to the packages' prepare scripts to the parent package.json prepare script, which was a good enough workaround for me.

[nkbt]

Some more findings (yarn workspaces + lerna) that might be helpful

With workspaces lerna bootstrap is unnecessary and does nothing, so we just run prepublish after initial install

yarn install --pure-lockfile
yarn lerna run prepublish

Lerna is smart enough to run prepublish in correct order so libs are not getting build before their deps.

The only thing is you should not enforce parallel lerna build like yarn lerna run prepublish -- --parallel. In this case you may get build errors when order matters.

[jquense]

I've also found the prepublish lerna approah works well. However the one problem with the lerna run approach is packages that define binaries that other packages use in their prepublish scripts. Lerna runs them in the correct order, but because the the prepublish creates the bin file after the install process is run none of the other local packages have that bin linked in thier .bin folders and so break.

[Hypnosphi]

@jquense my workaround was to create additional entries that just require the actual binaries. Those entries are present at the moment of installation, so yarn has something to link. See https://github.com/storybooks/storybook/pull/1810/commits/4d10a559978c5e1d1391b022e7110576c5c71a0d#diff-6c1388595b10c4ae5635b68cf08edc98

[darkobits]

This behavior is currently making it impossible to migrate from Lerna's bootstrap to Yarn Workspaces.

It is imperative that the prepare/prepublish scripts are run (and in the correct order in each package, based on the dependency graph) to ensure that linked dependencies are built and available to dependents.

[bestander]

Sounds reasonable, send a PR

[martijnthe]

As a work-around, what seems to work for us is to add the lifecycle scripts to the root package.json and use lerna to "forward" them to the packages in the workspace, so for example:

repo_root/package.json:

"scripts": {
  "prepare": "lerna run --stream --sort prepare"
  ... etc ...
}

If yarn install is run in the root, prepare will get run for all the packages in the workspace.

[martijnthe]

Incidentally, I'm now running into: https://github.com/yarnpkg/yarn/issues/4973 Is that the same issue perhaps?

The work-around I had posted above ^^^^ doesn't work for packages that aren't part of the workspace.

[stipsan]

Is anybody working on this? If not I'll give it a shot 😄

[bestander]

Go for it

[ccapndave]

For my (very simple) use-case I need a top-level prepare to run prepare in a specific workspace, so in the root package.json I added:

{
  "scripts": {
    "prepare": "yarn workspace my-workspace run prepare"
  }
}

Not very pretty, but gets the job done for the moment.

https://www.npmjs.com/package/wsrun might also be useful.

[FezVrasta]

I'm sorry to bump this but is anyone actually working on this? Workspaces are kinda half baked until this feature is implemented.

[stipsan]

I haven't had time to start on it yet 👨‍💻

[stipsan]

I hope I don't deter anyone else from starting, so unless I update this thread and say I'm working on it please assume that I'm not working on it 😅

[NathanielHill]

I'm using @ccapndave technique to trigger prepare in a shared react component library that needs transpilation. Would be nice if #6869 could be merged, even if it's only for prepare and prepublish as there are ordering issues with the others.

[thejohnfreeman]

I'm using @martijnthe's solution above and when I yarn, the top-level prepare script runs twice. When I yarn prepare, it only runs once, as expected. Does this happen to anyone else?

  "scripts": {
    "prepare": "lerna run prepare"
  }

[FezVrasta]

For everybody in need of this feature. Take a look at lerna-alias. It's a good way to avoid to need to build the workspace packages during developement

[mike-marcacci]

Came here looking for a different problem, but wanted to note that I don't terribly mind the current state of things, as it allows me to control which packages' scripts get run. Imagine a really, really big monorepo: you wouldn't necessarily want to run every prepare script just to work on one!

If you want to simulate the proposed behavior, you can quite easily do so in the corresponding lifecycle script in the top-level package.json:

{
  // ...
  "scripts": {

    // This runs the `prepare` script in each package.
    "prepare": "for P in packages/*/; do echo \"\n\n---------- $P\" && (cd $P && yarn prepare); done"

    // This runs the `test` script in each package, but exits on the first failure
    "test": "for P in packages/*/; do echo \"\n\n---------- $P\" && if ! (cd $P && yarn test); then exit 1; fi; done"
  }
}

I think any solution that changes the current behavior would need a way to opt out.

[pladaria]

@mike-marcacci if you want to avoid building already built packages, then skip build process if already built

yarn should execute all lifecycle scripts because you don't know what dependencies your package has. The whole repo must be built.

[mike-marcacci]

Yes, I think that there is probably a better strategy than an all-or-nothing approach to this (regardless of what the default behavior is). Let's say I have a repo with 26 packages – I may want to work in packages/c which depends on packages/b which depends on packages/a. I would ideally be able to run yarn inside packages/c, and yarn would know to:

1. install all external dependancies
2. run the prepare script in packages/a
3. run the prepare script in packages/b
4. run the prepare script in packages/c

...without building the remaining 23 packages.

packages/a/package.json
packages/b/package.json
packages/c/package.json
...
packages/z/package.json

package.json

Additionally, running yarn from the project root would be able to traverse the dependency tree and build all packages in the correct order. Without this, running all prepare scripts would be useless in many scenarios (since they may be run out-of-order).

AuthX is one project I've been working on where the build order is important. Note how the build script hard-codes the order of some packages, which are dependancies of the later ones.

[dosentmatter]

The root prepare script is still called during yarn install, so you can take advantage of that to call prepare in each of your workspaces.

Easiest way - using yarn workspaces with lerna: This will run prepare in all workspaces that contain a prepare script. Make sure your dependency graph is correct so lerna can run the prepare scripts in the correct order - the default for lerna is topological (least dependent package runs first).

# package.json
  "scripts": {
    "prepare": "lerna run prepare",
    ...
  },

If you are using yarn and yarn workspaces by itself: This solution can work, but has a negative where workspaces that do not contain a prepare script will still have prepare run and you will see error Command "prepare" not found. Your yarn install or yarn prepare will exit right at that moment. A workaround is to add a prepare script to all workspaces, even if it's just a NOP.

# package.json
  "scripts": {
    "prepare": "yarn workspaces run prepare",
    ...
  },

Or you can specifically select which workspaces will have prepare run:

# package.json
  "scripts": {
    "prepare": "yarn workspace workspace-a run prepare && yarn workspaces workspace-b run prepare",
    ...
  },

Or you can write a shell script that parses the JSON output from yarn workspaces info. Then you can run yarn workspace $workspace_name run prepare for each workspace. This way, you can ignore errors if the prepare script doesn't exist. I don't think yarn has a node library. But you can call it from node using shelljs for easy JSON parsing.

# package.json
  "scripts": {
    "prepare": "./scripts/prepare.sh",
    ...
  },

[ryanhiebert]

yarn workspaces run prepare would be a nice solution for me if I could ensure that sub-dependencies are prepared first. For example, I have package a that depends on packages b and c, and c depends on d. It's important, then, that that d run before c, and b and c both run before a. Apart from that, I don't care what order they run in. It would be really, really nice if yarn would do this for me, ideally, or secondarily give me a command that will list all of the workspaces in a dependecy-deterministic order for this purpose. Parsing these inter-dependencies seems well beyond the scope of what I should be doing in order to write a simple prepare script.

[dosentmatter]

@ryanhiebert, is there any reason you don't want to use lerna? It handles the dependency order for you and runs them concurrently. It's also not difficult to add to your project and integrate with yarn workspaces.

How does it [yarn] compare to Lerna?

Yarn’s workspaces are the low-level primitives that tools like Lerna can (and do!) use. They will never try to support the high-level feature that Lerna offers, but by implementing the core logic of the resolution and linking steps inside Yarn itself we hope to enable new usages and improve performance.

I don't think it's worth the development time for yarn to support all the things that lerna run can already do.

[itsMapleLeaf]

is there any reason you don't want to use lerna? It handles the dependency order for you and runs them concurrently. It's also not difficult to add to your project and integrate with yarn workspaces.

How does it [yarn] compare to Lerna?

Yarn’s workspaces are the low-level primitives that tools like Lerna can (and do!) use. They will never try to support the high-level feature that Lerna offers, but by implementing the core logic of the resolution and linking steps inside Yarn itself we hope to enable new usages and improve performance.

I don't think it's worth the development time for yarn to support all the things that lerna run can already do.

I'd prefer not to have to pick up, learn, and manage an entire second tool that does a lot more than I need just for one feature. But I can't speak for others.

[ecanchev-jaspersoft]

I cannot use preinstall and put a lerna command because the lerna package is not downloaded when executing the preinstall. What is the alternative here?

Logs:

2024-05-23 11:51:28.868 | . yarn install v1.22.19
[2](pipeline-console/?start-byte=0&selected-node=84#log-2)
2024-05-23 11:51:28.868 | . $ lerna run --stream --parallel preinstall
[3](/pipeline-console/?start-byte=0&selected-node=84#log-3)
2024-05-23 11:51:28.868 | . /bin/sh: lerna: command not found
[4](/pipeline-console/?start-byte=0&selected-node=84#log-4)
2024-05-23 11:51:28.868 | . error Command failed with exit code 127.
[5](/pipeline-console/?start-byte=0&selected-node=84#log-5)
2024-05-23 11:51:28.868 | . info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
[6](/pipeline-console/?start-byte=0&selected-node=84#log-6)
script returned exit code 127

the 'prepare' script is not an option for me as I want to execute a script before the installation of the packages. I'm coming from lerna bootstrap and want to adopt the workspaces provided by yarn. yarn version: 1.22.x