Cannot upgrade to 2.x: off-sync `package.json` and `package-lock.json`

fedenusy commented 8 months ago

Stems from an nx problem: https://github.com/nrwl/nx/issues/17653

Happens anytime I deploy functions (functions build ok).

Any known workarounds? Or just need to wait for upstream fix before upgrading to 2.x?

simondotm commented 8 months ago

@fedenusy Are you seeing this problem when running npm ci on the workspace root? or something else?

Deploying functions should not need any type of npm ci, nor a package-lock.json in the dist folder for a built function to deploy successfully.

simondotm commented 8 months ago

Also, are you using v1 or v2 cloud functions?

fedenusy commented 8 months ago

@simondotm npm ci on the workspace root works fine, as does npx nx build firebase-functions. The problem comes up when I run npx nx deploy firebase-functions.

I am not modifying dist/ in any way, just letting the plugin do its thing.

Using v2 cloud functions.

simondotm commented 8 months ago

Ok - some things that would be helpful to see what the problem could be:

Just to confirm - you are migrating a workspace from v1.x of the nx-firebase plugin to v2.x?
What is your nx version?
Could you share the firebase deploy errors that you are seeing?

Thanks

fedenusy commented 8 months ago

Correct, migrating from v1.x of the plugin to v2.x. Doing so by generating a new firebase v2 application and shifting a single function into the new app. (Also migrating that function from cloud functions v1 to cloud functions v2 in the process.)

Using nx 16.10.0.

Logs:

npx nx deploy firebase-functions                                                                                                        ✘ 1 f/cloud-functions-v2 ✱

   ✔    6/6 dependent project tasks succeeded [5 read from cache]

   Hint: you can run the command with --verbose to see the full dependent project outputs

 ————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————

> nx run firebase-functions:build  [local cache]

> nx run firebase-functions:deploy

 >  NX   Running target deploy for project firebase and 9 tasks it depends on:

> nx run email:handlebars-precompile
> nx run firebase:build  [local cache]
Build succeeded.
> nx run firebase:upload-sentry-sourcemaps
Done
> nx run firebase:deploy --only functions:firebase-functions
> nx run firebase:firebase deploy --only functions:firebase-functions
=== Deploying to 'my-firebase-project'...
i  deploying functions
i  functions: preparing codebase firebase-functions for deployment
i  functions: ensuring required API cloudfunctions.googleapis.com is enabled...
i  functions: ensuring required API cloudbuild.googleapis.com is enabled...
i  artifactregistry: ensuring required API artifactregistry.googleapis.com is enabled...
✔  artifactregistry: required API artifactregistry.googleapis.com is enabled
✔  functions: required API cloudfunctions.googleapis.com is enabled
✔  functions: required API cloudbuild.googleapis.com is enabled
i  functions: Loading and analyzing source code for codebase firebase-functions to determine what to deploy
Serving at port 8206
shutdown requested via /__/quitquitquit
i  functions: Loaded environment variables from .env, .env.my-firebase-project.
i  functions: preparing dist/packages/firebase-functions directory for uploading...
i  functions: packaged /Users/fedenusy/dev/farmpro/dist/packages/firebase-functions (163.71 KB) for uploading
i  functions: ensuring required API run.googleapis.com is enabled...
i  functions: ensuring required API eventarc.googleapis.com is enabled...
i  functions: ensuring required API pubsub.googleapis.com is enabled...
i  functions: ensuring required API storage.googleapis.com is enabled...
✔  functions: required API eventarc.googleapis.com is enabled
✔  functions: required API run.googleapis.com is enabled
✔  functions: required API pubsub.googleapis.com is enabled
✔  functions: required API storage.googleapis.com is enabled
i  functions: generating the service identity for pubsub.googleapis.com...
i  functions: generating the service identity for eventarc.googleapis.com...
✔  functions: dist/packages/firebase-functions folder uploaded successfully
i  functions: creating Node.js 18 (2nd Gen) function firebase-functions:myTrigger(us-central1)...
Build failed with status: FAILURE and message: npm ERR! code EUSAGE
npm ERR!
npm ERR! `npm ci` can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.
npm ERR!
npm ERR! Missing: color-convert@2.0.1 from lock file
npm ERR! Missing: color-name@1.1.4 from lock file
npm ERR! Missing: kind-of@3.2.2 from lock file
npm ERR! Missing: kind-of@3.2.2 from lock file
npm ERR! Missing: kind-of@6.0.3 from lock file
npm ERR! Missing: color-convert@2.0.1 from lock file
npm ERR! Missing: supports-color@7.2.0 from lock file
npm ERR! Missing: color-name@1.1.4 from lock file
npm ERR! Missing: ansi-styles@4.3.0 from lock file
npm ERR! Missing: color-convert@2.0.1 from lock file
npm ERR! Missing: color-name@1.1.4 from lock file
npm ERR! Missing: ansi-styles@4.3.0 from lock file
npm ERR! Missing: color-convert@2.0.1 from lock file
npm ERR! Missing: color-name@1.1.4 from lock file
npm ERR! Missing: supports-color@7.2.0 from lock file
npm ERR! Missing: has-flag@4.0.0 from lock file
npm ERR! Missing: kind-of@6.0.3 from lock file
npm ERR! Invalid: lock file's domhandler@3.3.0 does not satisfy domhandler@4.3.1
npm ERR! Invalid: lock file's domhandler@4.3.1 does not satisfy domhandler@3.3.0
npm ERR!
npm ERR! Clean install a project
npm ERR!
npm ERR! Usage:
npm ERR! npm ci
npm ERR!
npm ERR! Options:
npm ERR! [--install-strategy <hoisted|nested|shallow|linked>] [--legacy-bundling]
npm ERR! [--global-style] [--omit <dev|optional|peer> [--omit <dev|optional|peer> ...]]
npm ERR! [--strict-peer-deps] [--foreground-scripts] [--ignore-scripts] [--no-audit]
npm ERR! [--no-bin-links] [--no-fund] [--dry-run]
npm ERR! [-w|--workspace <workspace-name> [-w|--workspace <workspace-name> ...]]
npm ERR! [-ws|--workspaces] [--include-workspace-root] [--install-links]
npm ERR!
npm ERR! aliases: clean-install, ic, install-clean, isntall-clean
npm ERR!
npm ERR! Run "npm help ci" for more info
npm ERR! A complete log of this run can be found in: /www-data-home/.npm/_logs/2023-10-14T13_18_32_458Z-debug-0.log. For more details see the logs at https://console.cloud.google.com/cloud-build/builds;region=us-central1/1ba8ffc7-9fb9-41e1-b66d-65868af6ef4c?project=195657483410.
Functions deploy had errors with the following functions:
    firebase-functions:myTrigger(us-central1)
i  functions: cleaning up build files...
Error: There was an error deploying functions
Warning: run-commands command "firebase --config=firebase.json deploy --only functions:firebase-functions" exited with non-zero status code

 >  NX   Running target firebase for project firebase failed
   Failed tasks:

   - firebase:firebase

   Hint: run the command with --verbose for more details.
Warning: run-commands command "nx run firebase:firebase deploy --only functions:firebase-functions" exited with non-zero status code

 >  NX   Running target deploy for project firebase and 9 tasks it depends on failed
   Failed tasks:

   - firebase:deploy

   Hint: run the command with --verbose for more details.
Warning: run-commands command "nx run firebase:deploy --only functions:firebase-functions" exited with non-zero status code

 ————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————————

 >  NX   Ran target deploy for project firebase-functions and 7 task(s) they depend on (1m)

    ✖    1/8 failed
    ✔    7/8 succeeded [6 read from cache]

simondotm commented 8 months ago

Ok thats great thanks. I understand the problem now and the relevance of the link you shared in your first post. So the issue can be described as:

We have a firebase function that uses various dependencies from the workspace root package.json
When we build this function with @nx/esbuild, Nx emits a package.json to the dist dir listing these dependencies, but it does not generate a package-lock.json
When firebase deploy runs, it determines dependencies by running npm ci, however due to the missing package-lock.json, peer dependencies may vary from the workspace

simondotm commented 8 months ago

I see that Nx have implemented generation of a pruned package-lock.json file when generatePackageJson build option is true, but it does not yet appear to be present for the @nx/esbuild plugin.

@fedenusy - could you try something for me?

in your functions project.json edit the deploy target from:

"deploy": {
      "executor": "nx:run-commands",
      "options": {
        "command": "nx run firebase:deploy --only functions:your-function"
      },
      "dependsOn": ["build"]
    }

to:

    "deploy": {
      "executor": "nx:run-commands",
      "options": {
        "commands": [
          "cp package-lock.json dist/apps/your-function/package-lock.json",
          "nx run firebase:deploy --only functions:your-function"
        ]
      },
      "dependsOn": ["build"]
    }

The idea here is that by copying the root package-lock.json to the dist folder before firebase deploys it should use the same package versions for npm ci.

simondotm commented 8 months ago

Also worth noting this isnt a migration v1.x -> v2.x issue. Its a v2.x / esbuild issue.

fedenusy commented 8 months ago

could you try something for me? ...

Thank you, the suggestion worked!

Also worth noting this isnt a migration v1.x -> v2.x issue. Its a v2.x / esbuild issue.

Agreed

simondotm commented 8 months ago

Raised a new ticket for this: https://github.com/simondotm/nx-firebase/issues/161

simondotm commented 6 months ago

Closing this issue, Nx 17+ seems to be generating package-lock.json files in dist correctly now.

simondotm / nx-firebase

Cannot upgrade to 2.x: off-sync `package.json` and `package-lock.json` #157