Individual lockfile per workspace

[migueloller]

• [x] I'd be willing to implement this feature
• [ ] This feature can already be implemented through a plugin

Describe the user story

I believe the best user story are the comments in this issue: https://github.com/yarnpkg/yarn/issues/5428

Describe the solution you'd like

Add support for generating a lockfile for each workspace. It could be enabled via configuration option like lockfilePerWorkspace?

Describe the drawbacks of your solution

As this would be opt-in and follow well known semantics, I'm not sure there are any drawbacks to implementing the feature. One could argue if it should be in the core or implemented as a plugin. I personally don't have a preference of one over the other.

Describe alternatives you've considered

This could be implemented in the existing workspace-tools plugin. I'm unsure if the hooks provided by Yarn would allow for this, though.

Additional context

https://github.com/yarnpkg/yarn/issues/5428

[lukemovement]

Why not use a generic name for the lock files in individual workspaces, the same way package-lock.json is used as a fallback when yarn.lock is not present? This can then be generated after an install is ran from the root workspace.

/workspace 
yarn.lock
package.json
    - /packages
        - /my-app
        - package.json
        - yarn.workspace.lock

Executing yarn install from within /my-app when the parent workspace is not present would then result in yarn.workspace.lock being used as a fallback.

[zaro]

Here is my approach to this. Since we don't exactly have monorepo, but rather a repo with other repos as submodules this makes for quite problematic deployment strategy w/o individual lockfiles. So my solution is simply a fork of yarn-plugin-entrypoint-lockfiles but the lockfiles are generated next to the package.json with a fixed name yarn.deploy.lock. This makes Dockerfiles really simple. Also workspace: resolutions are replaced with actual version.

Link to the plugin https://github.com/zaro/yarn-plugin-deploy-lockfiles .

[asgeirn]

This is the workaround we use for a metarepository:

Meta root:

.yarnrc.yml
package.json
yarn-workspace.lock

Individual projects:

package.json
yarn.lock

Contents of root .yarnrc.yml:

lockfileFilename: yarn-workspace.lock

Running yarn install in the workspace root uses the yarn-workspace.lock file, as well as the workspaces definition and workspace:* resolutions in the workspace package.json file.

The individual projects are separate Git repositories and are built separately from each other using the local yarn.lock file.

There is one caveat - updating the per-project yarn.lock file. We have delegated this task to a GitHub action triggered as part of the pull request flow, running yarn install --mode=update-lockfile and committing an updated yarn.lock file.

[JasonMan34]

@arcanis Is there any plan to implement this in the future? The inability to have a lockfile (and individual .yarn folder) per workspace is such a huge downside it overshadows any upside using a workspace can provide us

[manoj-r]

I just started using Yarn Workspaces for some of my projects and have used nohoist/nmHoistLimits: workspaces, Does Yarn enforce the lock file to be generated only in the root repository even for noHoist? This is the behavior I see in my project. Can someone please confirm that the lock file will be generated only in the root project, regardless of the hoist configuration?

[lukemovement]

Yes that is correct. Hoisting only effects the physical files in the dependency project. Personally I use husky pre commit to copy yarn.lock from the workspace directory to the sub project folder and name it yarn.workspace.lock for the deployment scripts

On Thu, May 4, 2023, 18:47 Manoj @.***> wrote:

I just started using Yarn Workspaces for some of my projects and have used nohoist/nmHoistLimits: workspaces, Does Yarn enforce the lock file to be generated only in the root repository even for noHoist? This is the behavior I see in my project. Can someone please confirm that the lock file will be generated only in the root project, regardless of the hoist configuration?

— Reply to this email directly, view it on GitHub https://github.com/yarnpkg/berry/issues/1223#issuecomment-1535171372, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOIFDOLWREDQCIBTG2M3STXEPTR5ANCNFSM4MMVPXLA . You are receiving this because you commented.Message ID: @.***>

[bertho-zero]

lockfileFilename no longer exists since v4 and it breaks the solutions that some people have built here

[borekb]

True what @bertho-zero said above ☝️, unfortunately we can't upgrade to Yarn 4 because of the removal of lockfileFilename in https://github.com/yarnpkg/berry/pull/5604.

I understand why the team wanted to remove it but there's no other solution to the use cases described in this issue that I would be aware of.

[arcanis]

I'm interested to try to figure out a proper integrated solution for the 4.1. Would it solve your use cases if yarn workspaces focus was updating the lockfile to remove the dependencies from unfocused packages?

[borekb]

I'm not quite sure. Separate lockfiles, as implemented by https://github.com/JanVoracek/yarn-plugin-entrypoint-lockfiles, are admittedly a bit wasteful (the contents of lockfiles is partly duplicated and commands like yarn add are slower) but they also have several advantages, like being simple to reason about, our Docker build is straightforward (we pass a file like yarn.docusaurus.lock to the build context), we can view history of separate lockfiles in Git, we can trigger GitHub Actions workflows based on changes in those separate lockfiles (for example, rebuild our docs image if dependencies slightly changed), our scripts can calculate the "latest Git commit touching anything related to our Docusaurus docs" which should include yarn.docusaurus.lock but couldn't have included yarn.lock because it is shared, etc.

It's hard for me to imagine how to support all of this (and those are real use cases BTW) with a single shared lockfile. I have to admit that the idea of separate lockfiles was quite controversial in our team initially and it's still relatively weird to see several yarn.<something>.lock files in our repo but on the upside, it handles every use case we threw at it well.

[borekb]

BTW it's great that you're thinking how to implement this for 4.1!

[valleywood]

We are also having an issue with this and it prevents us from stepping up to Yarn 4 as we have a workspace solution that without lockfileFilename will lead to conflicting lockfiles. @arcanis sounds great if you could figure out an alternative way of solving this now when the lockfileFilename option has been removed! 🙏

The use case is that we are having a yarn workspace containing a number of npm package repos. W hen running yarn install on one of these packages (having a yarn.lock lockfile in the workspace root) the yarn.lock file generated in the workspace context will differ from what the lockfile would have looked like if yarn install was run in an environment when the package code is located outside the workspace.

If we commit this lockfile when running yarn install inside the workspace it will be a problem when running yarn install with a frozen lockfile as a part of our CI/CD process with our GitHub actions as that will throw an error as it will be indicating that we are trying to modify the lockfile as it is then generated outside the workspace and wont have the sam structure.

Being able to rename the lockfile in the workspace directory solved this so that the yarn.lock file in each subrepo was unaffected by running in a workspace environment thereby causing no issues running yarn install with frozen lockfile in the CI/CD processes of each subrepo.

Example Works (yarn lockfile generated in subrepo is the same when running yarn install inside/outside the workspace)

Workspace lockfile                  Sub-repo lockfile           Sub-repo CI/CD lockfile
yarn-workspace.lock                 yarn.lock                   yarn.lock

Doesn't work (yarn lockfile generated in subrepo differs when running yarn install inside/outside the workspace)

Workspace lockfile                  Sub-repo lockfile           Sub-repo CI/CD lockfile
yarn.lock                           yarn.lock                   yarn.lock

[trusktr]

My workspaces are also git submodules in my super repo (for example). Any of the workspaces can be cloned separately on their own. I'd like those workspaces to have their own lock files for when they are cloned separately. The workspace lockfiles would be ignored when installing using the top-level workspace, of course (the top level would ensure they are in sync, and could even throw an error if they are not in sync, and maybe even provide an option to force-overwrite the workspace lock file to fix any issue)

[jakebailey]

@trusktr FWIW what you're describing is what I implemented in https://github.com/jakebailey/yarn-plugin-workspace-lockfile via a plugin (modified from someone else's attempt); I ended up not needing it personally (the team I was on didn't end up adopting yarn), but it did seem to work well a the time those many years ago. It's possible it still works, or could with modification for Yarn 4.

[FezVrasta]

Would a pnpm deploy equivalent work to cover this use case?