search

0x80 / isolate-package

Isolate a monorepo package with its internal dependencies to form a self-contained directory with a pruned lockfile
MIT License
121 stars 13 forks source link

Yarn v3: The lockfile would have been modified by this install, which is explicitly forbidden. #20

Closed darkmirage closed 10 months ago

darkmirage commented 1 year ago

Thanks for making this package because it's been sorely needed for a long time for FIrebase Functions deployment. I have most of it working based on your instructions, but yarn install --immutable is breaking on deployments.

Error

In the below example, @arc/feature-flag is a workspace that was relocated to functions/isolate/modules/feature-flag from modules/feature-flag by yarn isolate. The error shows up when we attempt to deploy to firebase functions.

Logs from Google Cloud Build triggered by Firebase deploy:

INFO 2023-08-21T22:16:21.352014524Z Step #2 - "build": Installing Yarn v3.6.1
INFO 2023-08-21T22:16:21.352389680Z Step #2 - "build": 2023/08/21 22:16:21 [DEBUG] GET https://repo.yarnpkg.com/3.6.1/packages/yarnpkg-cli/bin/yarn.js
INFO 2023-08-21T22:16:21.616034608Z Step #2 - "build": DEBUG: Setting environment variable PATH=/layers/google.nodejs.yarn/yarn_engine/bin:/layers/google.nodejs.runtime/node/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
INFO 2023-08-21T22:16:21.660331912Z Step #2 - "build": WARNING: Skipping adding auth token to /www-data-home/.yarnrc.yml. Unable to find Artifact Registry in /workspace/.yarnrc.yml.
INFO 2023-08-21T22:16:21.660349522Z Step #2 - "build": --------------------------------------------------------------------------------
INFO 2023-08-21T22:16:21.660351700Z Step #2 - "build": Running "yarn install --immutable"
INFO 2023-08-21T22:16:22.540249844Z Step #2 - "build": ➤ YN0000: ┌ Resolution step
INFO 2023-08-21T22:16:23.150742724Z Step #2 - "build": ➤ YN0013: │ @arc/feature-flag@file:./modules/feature-flag#./modules/feature-flag::hash=7122b8&locator=%40arc%2Ffunctions%40workspace%3A. can't be found in the cache and will be fetched from the disk
INFO 2023-08-21T22:16:23.310145669Z Step #2 - "build": ➤ YN0000: └ Completed in 0s 769ms
INFO 2023-08-21T22:16:23.518625341Z Step #2 - "build":
INFO 2023-08-21T22:16:23.519048711Z Step #2 - "build": ➤ YN0000: ┌ Post-resolution validation
INFO 2023-08-21T22:16:23.519058767Z Step #2 - "build": ➤ YN0000: │ @@ -217,10 +217,9 @@
INFO 2023-08-21T22:16:23.519060244Z Step #2 - "build": ➤ YN0000: │ linkType: hard
INFO 2023-08-21T22:16:23.519062243Z Step #2 - "build": ➤ YN0000: │
INFO 2023-08-21T22:16:23.519063426Z Step #2 - "build": ➤ YN0000: │ "@arc/feature-flag@file:./modules/feature-flag::locator=%40arc%2Ffunctions%40workspace%3A.":
INFO 2023-08-21T22:16:23.519072723Z Step #2 - "build": ➤ YN0000: │ version: 1.0.0
INFO 2023-08-21T22:16:23.519751074Z Step #2 - "build": ➤ YN0028: │ - resolution: "@arc/feature-flag@file:./modules/feature-flag#./modules/feature-flag::hash=7b4bc8&locator=%40arc%2Ffunctions%40workspace%3A."
INFO 2023-08-21T22:16:23.519999320Z Step #2 - "build": ➤ YN0028: │ - checksum: 9697a40672390184a2b0cc3d42f437a1e56e9e63b3fc24658793cdacec3681b95a373d13bc856b90b1cb9c2ca7cb337b4d2916c9cdaae5ddf6ae808e8fdec42f
INFO 2023-08-21T22:16:23.520005499Z Step #2 - "build": ➤ YN0028: │ + resolution: "@arc/feature-flag@file:./modules/feature-flag#./modules/feature-flag::hash=7122b8&locator=%40arc%2Ffunctions%40workspace%3A."
INFO 2023-08-21T22:16:23.520006863Z Step #2 - "build": ➤ YN0000: │ languageName: node
INFO 2023-08-21T22:16:23.520298259Z Step #2 - "build": ➤ YN0000: │ linkType: hard
INFO 2023-08-21T22:16:23.520474017Z Step #2 - "build": ➤ YN0000: │
INFO 2023-08-21T22:16:23.520479605Z Step #2 - "build": ➤ YN0000: │ "@arc/functions@workspace:.":
INFO 2023-08-21T22:16:23.520945843Z Step #2 - "build": ➤ YN0000: │
INFO 2023-08-21T22:16:23.521661955Z Step #2 - "build": ➤ YN0028: │ The lockfile would have been modified by this install, which is explicitly forbidden.
INFO 2023-08-21T22:16:23.521782341Z Step #2 - "build": ➤ YN0000: └ Completed in 0s 205ms
INFO 2023-08-21T22:16:23.522244724Z Step #2 - "build": ➤ YN0000: Failed with errors in 0s 987ms

Setup

isolate.config.json:

{
  "includeDevDependencies": true,
  "targetPackagePath": "./functions",
  "workspaceRoot": "./"
}

firebase.json:

{
  "functions": {
    "source": "functions/isolate",
    "ignore": [
      "**/.yarn/cache/**",
      "**/.yarn/install-state.gz",
      "**/.git",
      "**/.runtimeconfig.json",
      "**/firebase-debug.log",
      "**/firebase-debug.*.log",
      "**/node_modules/**"
    ]
  }
}

Pre-deploy steps in CI job

yarn isolate
cp -r .yarn ./functions/isolate/.yarn
cp .yarnrc.yml ./functions/isolate/.yarnrc.yml
cd ./functions/isolate

# On hindsight this probably does not do anything because Cloud Build explicitly 
yarn config set enableImmutableInstalls falseruns yarn install --immutable
yarn install --mode=update-lockfile 

yarn firebase deploy -P $FIREBASE_PROJECT --only functions --force --debug
0x80 commented 1 year ago

I'm not sure why you'd want to have these manual copy steps. I think the isolate command should handle everything if you configure it as part of the firebase.json "predeploy" configuration.

That being said there seems to be an issue with the yarn lockfile still, preventing it to be use in the deployment as-is, so I advise you to exclude the lockfile using the configuration settings.

For more info see:

https://github.com/0x80/isolate-package/issues/12

https://github.com/0x80/isolate-package/#lockfiles

0x80 commented 1 year ago

I'm hoping to find some time soon to really get deeper into the lockfile formats for PNPM and Yarn and see if it's possible to modify them to make the deployments work, but it might be difficult to achieve and I struggle to find time at the moment...

None of the manual workarounds solve this issue, and it's not a deal breaker for me, but I guess it depends on your situation and the kind of dependencies you deploy.

A temporary solution could maybe be to use exact versions in your package.json file. It doesn't lock the dependencies of your dependencies, like a lockfile would, but at least the direct dependencies of your project will be the same...

darkmirage commented 1 year ago

I did end up having to not include the lockfile for Firebase Functions deploy. Haven't run into problems with it for now but we plan to migrate away from Firebase Functions anyway.

The reason I was doing the manual copies was because some of the cache and config file live outside of the targetPackagePath. In our Yarn monorepo, the individual workspace /functions has its Yarn artifacts hoisted up to /. I don't think isolate is accounting for that as far as I know. Both .yarn/releases and .yarnrc.yml` are required to configure Yarn to run correctly.

I believe the issue with the lockfile is that the hash for a local file: reference was not a stable identity after the upload to Firebase Functions and Cloud Build. I'm not sure how Yarn is generating that hash.

0x80 commented 10 months ago

@darkmirage I'm happy to report that lockfile output for classic and modern version of Yarn is now supported in the latest version. At least I've tested it with v1 and v4.

For v4 it does not generate a yarn.lock file, but instead for modern yarn versions it falls back to using NPM. The generated packge-lock in the output is based on the installed node_modules and therefore should match all the versions of your original yarn lockfile. For more info see lockfiles

What package manager is used in the deployment probably shouldn't matter (as long as the locked versions match) so I think this is a viable solution.

I am also working on a typescript monorepo boilerplate which has a branch called use-yarn-modern that you can check out if you want to see a working example with isolate-package integrated.