laurenzlong commented 6 years ago

See: https://github.com/firebase/firebase-functions/issues/172

Version info

3.17.3

Steps to reproduce

Expected behavior

Actual behavior

mesqueeb commented 3 years ago

it worries me that there is no google employee assigned to this issue though. Does anyone have some insight?

M3kH commented 3 years ago

Is a bit demotivating this issue is open since 2018. I would be glad to help if I can but of course we need some assistance to understand which code runs on the firebase function codebase to proper commit meaningful changes on that codebase.

johnnyoshika commented 3 years ago

Well that was an interesting challenge. I use NPM Workspaces for my monorepo setup with TypeScript and GitHub workflows for CI. Here's how I got it to work:

Build my dependent packages using TypeScript's tsc
Copy the compiled files into the functions folder (functions/src/domain to be exact) so that they get uploaded by Firebase CLI
Insert local dependency inside functions/package.json. I'm doing it with sed, which GitHub actions supports:

sed -i 's/"dependencies": {/"dependencies": { "@mycompany\/domain": "file:src\/domain",/' package.json

Before that command, my package.json looks like this:

  "dependencies": {
    "lodash": "^4.17.20",
  ...

After that command, it looks like this:

  "dependencies": { "@mycompany/domain": "file:src/domain",
    "lodash": "^4.17.20",
  ...

Then deploy: firebase deploy --only functions

simondotm commented 3 years ago

In case it's a useful reference for anyone, I recently created a firebase plugin for Nx that mainly takes advantage of the firebase CLI --config option to allow multiple firebase projects to co-exist in an Nx mono-repo, each with their own firebase.<projectname>.json file.

For functions deployment, where there are shared local libraries being imported, I do the same as @johnnyoshika - first copy the built dependent libraries into whatever output/dist folder you have for your functions, so that all functions code is in one place, and then use local package references to these libs in the functions package.json. I've written up some notes in the readme here.

I keep all of the firebase configs in the root of my workspace, so the firebase CLI works as usual, but I keep the project specific resources (rules/indexes etc.) within my firebase functions "app" in the mono repo workspace; the root firebase configs simply point to these files in the relevant directory.

Generally (imho) I'd recommend steering clear of using webpack for functions. There's no compelling reason afaik to bundle/minify functions code, and furthermore it prevents use of dynamic function export techniques for optimizing cold starts.

What I'd like to figure out next with multiple firebase projects in a mono-repo is how to protect against accidental deploys between project aliases, since in a mono-repo setup like this there's only one .firebaserc with multiple targets in it, and its scarily easy to firebase use the wrong project and deploy the wrong functions to it... so I've commented on that here.

hartherbert commented 3 years ago

Hey @johnnyoshika I do the same but since I moved to npm workspaces (was using yarn before) I get the error that my package-lock.json file is not in sync with my package.json file, since I updated it with the local dependencies

Here is the error while I am running firebase deploy --only functions:

Build failed: npm ERR! cipm can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.
npm ERR! 
npm ERR! 
npm ERR! Invalid: lock file's @mono-repo/shared@1.3.4 does not satisfy @mono-repo/shared@file:./src/local-dependencies/shared
npm ERR! Invalid: lock file's @mono-repo/shared-config@1.3.4 does not satisfy @mono-repo/shared-config@file:./src/local-dependencies/shared-config
npm ERR!

I also need to mention that I have 3 packages in my monorepo shared, shared-config and shared-admin. Also shared-admin has a dependency on shared and shared-config.

Before the deploy starts I copy all three packages to my functions/src/local-dependencies. This worked with yarn and yarn.lock but is not working with npm and package-lock.json

Any idea how to resolve this problem? Any hint would be awesome.

johnnyoshika commented 3 years ago

@hobbydevs That is very strange. I haven't come across that problem and I have a fairly large monorepo. What functions rules (e.g. predeploy rules) do you have set up in your firebase.json file?

johnnyoshika commented 3 years ago

@hobbydevs I just thought of something. Do you have a package-lock.json in your functions folder? You shouldn't have one there in your monorepo, and all of your npm install should be done at the root level. There should only be one package-lock.json at the root. So when you deploy your functions folder, package-lock.json shouldn't be sent to GCP.

simondotm commented 3 years ago

I also do not create a package-lock.json for my functions deployment in our monorepo setup. I squinted at the firebase docs that suggest node 12 engine requires it, but that (thankfully) does not seem to be enforced. Other firebase docs also seem to concur its optional:

Note: If a package-lock.json or yarn.lock file is found within your project, that lock file will be respected when dependencies are installed using npm ci or yarn install.

hartherbert commented 3 years ago

I managed to make it work by also adding the .tgz dependency to my @mono-repo/shared-admin package. This was throwing the error since the shared-admin package required those other two packages.

Also I only have one package-lock.json at root level.

My process looks roughly like this

build all sub-packages
pack each of them npm pack
Install those .tgz packages to my functions and also to the shared-admin package
lint functions
build functions
deploy it

This whole process is managed by a script I wrote.

KabutoYamato commented 3 years ago

maybe something like rollup could help to pack the shared dependencies inside the functions code, in rollup if you dont write a module dependency as external it will try to pack it with the code

serhiipalash commented 2 years ago

Thanks to everyone for your ideas! Let me share my solution of this problem. I hope it can be useful.

First of all, the only problem is in one step in GCloud build workflow which runs yarn install. Having Yarn Workspaces monorepo doesn't allow this step to finish as not all (or maybe none) of functions descendent workspaces (dependencies) are available in GCloud as only functions directory is uploaded. yarn install works fine with monorepo workspaces, it's just the workspaces are missing (not uploaded to GCloud.

In short, the solution I made is to mock monorepo Yarn Workspaces inside functions /dist directory, by copying descendent workspaces, root yarn.lock and adding "workspaces" field in functions package.json. All this is eliminated after deploy is finished.

In details

Our monorepo structure looks like this

/
    yarn.lock
    /libs
        /utils
            /src
            tsconfig.json
            package.json
    /apps
        /functions
            /scripts
                deploy-tasks.js
            /src
            package.json
            tsconfig.json
            firebase.json

/apps/functions/firebase.json

{
  "functions": {
    "source": "."
  }
}

/apps/functions/package.json

{
  "main": "dist/app/index.js",
  "scripts": {
    "build": "tsc -b",
    "predeploy": "gulp --gulpfile ./scripts/deploy-tasks.js predeploy",
    "deploy": "firebase deploy --only functions",
    "postdeploy": "gulp --gulpfile ./scripts/deploy-tasks.js postdeploy"
  },
  "devDependencies": {
    "firebase-tools": "^9.8.0",
    "gulp": "4.0.2",
    "gulp-json-editor": "^2.5.6",
    "typescript": "4.4.3"
  }
}

/apps/functions/tsconfig.json

{
  "outDir": "./dist/app",
  ...
}

/libs/utils/tsconfig.json

{
  "outDir": "./dist",
  ...
}

The functions bundle lives under /apps/functions/dist/app/ and the shared utils bundle lives under /libs/utils/dist/.

The solution I did is

Predeploy phase

copy /apps/functions/package.json to /apps/functions/dist/
change "main" field in /apps/functions/dist/package.json to "app/index.js"
add "workspaces" field in /apps/functions/dist/package.json with all functions descendent workspaces
set "functions" source to "dist" in /apps/functions/firebase.json
copy yarn.lock from monorepo root to /apps/functions/dist/
copy /libs/utils/dist/ and /libs/utils/package.json to /apps/functions/dist/libs/utils/dist/ and /apps/functions/dist/libs/utils/package.json

Postdeploy phase

set "functions" source back to "." in /apps/functions/firebase.json. This is important to keep functions emulator not broken.

So, we don't deploy /apps/functions/ to GCloud, but /apps/functions/dist/ with its own package.json and /libs inside. When yarn install will run in /apps/functions/dist/, the /libs will be in place and the "workspaces" will be declared in package.json.

And here is the code of /apps/functions/scripts/deploy-tasks.js

const path = require('path')
const { src, dest, series } = require('gulp')
const jeditor = require('gulp-json-editor')

const workspacesInfo = getWorkspacesInfo()

const functionsWorkspaceName = process.env.npm_package_name

const functionsDescendentWorkspaces = Object.values(
  getDescendentWorkspaces({
    workspaceName: functionsWorkspaceName,
    workspacesInfo,
  })
)

const functionsWorkspace = workspacesInfo[functionsWorkspaceName]

const functionsPath = path.resolve(process.cwd(), '..')

const rootPath = functionsPath.replace(
  new RegExp(`/${functionsWorkspace.location}$`),
  ''
)

const paths = {
  root: rootPath,

  rootYarnLock: rootPath + '/yarn.lock',

  functions: functionsPath,

  functionsDist: functionsPath + '/dist',

  functionsPackageJson: functionsPath + '/package.json',

  firebaseJson: functionsPath + '/firebase.json',

  functionsDescendentWorkspaces: functionsDescendentWorkspaces.reduce(
    (acc, workspace) => {
      acc.push(
        `${rootPath}/${workspace.location}/dist/**/*`,
        `${rootPath}/${workspace.location}/package.json`
      )

      return acc
    },
    []
  ),
}

exports.predeploy = series(
  generateWorkspacesInFunctionsDist,
  setFunctionsSourceToDistInFirebaseJson
)

exports.postdeploy = setFunctionsSourceBackToFunctionsDirInFirebaseJson

function getWorkspacesInfo() {
  const { execSync } = require('child_process')

  const result = execSync('yarn workspaces info').toString()

  return JSON.parse(
    result.substring(result.indexOf('{'), result.lastIndexOf('}') + 1)
  )
}

function getDescendentWorkspaces({
  workspaceName,
  workspacesInfo,
  result = {},
}) {
  const workspace = workspacesInfo[workspaceName]

  if (!workspace) {
    throw new Error(`Missing info about "${workspaceName}" workspace`)
  }

  workspace.workspaceDependencies.forEach(dependencyWorkspaceName => {
    const dependencyWorkspace = workspacesInfo[dependencyWorkspaceName]

    result[dependencyWorkspaceName] = dependencyWorkspace

    if (dependencyWorkspace.workspaceDependencies.length) {
      getDescendentWorkspaces({
        workspaceName: dependencyWorkspaceName,
        workspacesInfo,
        result,
      })
    }
  })

  return result
}

function generateWorkspacesInFunctionsDist(cb) {
  src(paths.functionsPackageJson)
    .pipe(
      jeditor(json => {
        json.workspaces = functionsDescendentWorkspaces.map(
          workspace => workspace.location
        )

        json.main = json.main.replace(/^dist\//, '')

        return json
      })
    )
    .pipe(src(paths.rootYarnLock))
    .pipe(src(paths.functionsDescendentWorkspaces, { base: paths.root }))
    .pipe(dest(paths.functionsDist))

  cb()
}

function setFunctionsSourceToDistInFirebaseJson(cb) {
  src(paths.firebaseJson)
    .pipe(
      jeditor(
        {
          functions: {
            source: 'dist',
          },
        }
      )
    )
    .pipe(dest(file => file.base))

  cb()
}

function setFunctionsSourceBackToFunctionsDirInFirebaseJson(cb) {
  src(paths.firebaseJson)
    .pipe(
      jeditor(
        {
          functions: {
            source: '.',
          },
        }
      )
    )
    .pipe(dest(file => file.base))

  cb()
}

serhiipalash commented 2 years ago

If someone is interested, feel free to create a CLI util npm-package with my code. It's possible to make it an abstract, it requires only few arguments to work

{
  "scripts": {
    "predeploy": "firebase-functions-deploy-tools predeploy --firebaseJson ./firebase.json --functionsDir ./",
    "postdeploy": "firebase-functions-deploy-tools postdeploy --firebaseJson ./firebase.json"
  }
}

Also, the code and logic is not be so much complex and I think it's possible to integrate it directly to firebase-tools. Just not sure how to verify if you inside Yarn Workspaces monorepo or not. I guess the best is to provide a new config field in firebase.json for it.

Ping @samtstern

Stradivario commented 2 years ago

@serhiipalash

Have you try this

https://github.com/rxdi/firelink ?

0x80 commented 2 years ago

Interesting to see what people come up with, but I find using a bundler in the pre-deploy step with a few lines of configuration much easier to grasp and it hardly has an impact on your codebase. I wonder if I'm missing something 👀 🤔

hartherbert commented 2 years ago

@0x80 when you bundle your cloud functions they are not optimized for cold starts because everything needs to be loaded to run a single function. Depends on how many functions you have. This is what I read somewhere in the cloud functions docs.

serhiipalash commented 2 years ago

Have you try this

No, I haven't seen this tool. Thanks @Stradivario !

It looks very similar on what I did. I was thinking to solve the problem the same way you did, by editing functions package.json and then reverting it back after deploy, but then decided to not do it. I put all changes in /dist directory as everything is git-ignored then, and I only impact firebase.json (this can affect the source control) as this file is a deployment and local development config and not something about the app itself, which is not true about package.json.

0x80 commented 2 years ago

@hobbydevs Good to know! 👍

I don't deploy a huge amount of functions and I'm planning to move all my https triggered functions to Cloud Run or App Engine soon (*) so I think I'm good.

(*) Because I learned recently that https functions are not a great choice for creating an external API. Scaling issues make timeouts very likely to occur, so clients always need to use a retry mechanism. You can partially work around it by setting min_instances to something other than 0, but that will cost you of course.

Stradivario commented 2 years ago

@hobbydevs Good to know! 👍

I don't deploy a huge amount of functions and I'm planning to move all my https triggered functions to Cloud Run or App Engine soon (*) so I think I'm good.

(*) Because I learned recently that https functions are not a great choice for creating an external API. Scaling issues make timeouts very likely to occur, so clients always need to use a retry mechanism. You can partially work around it by setting min_instances to something other than 0, but that will cost you of course.

You may research patterns like Saga, Event sourcing. These are one of the most efficent and resilent applications.

Triggering lambda using http has an overhead and it is a synchronious in terms of the webserver execution.

0x80 commented 2 years ago

@Stradivario My https functions are serving an API to external clients. I don't understand what your patterns have to do with it. The reason I'm moving them to Cloud Run or App Engine is because of scaling behavior and availability. Those platforms seem to be more suitable for hosting APIs.

johnnyoshika commented 2 years ago

I've been really happy with the solution I posted above direct link. It requires my package.json to be modified, but only in my CI environment (I use GitHub Actions but any CI will work). The CI environment is ephemeral anyways, so the modified package.json is blown away once deployment is done. It only requires a few lines of code in my YML file. I've been using the technique for 5 months and I haven't had to think about it since I implemented it. If anyone wants more information, I'd be happy to help.

vthommeret commented 2 years ago

This may not apply to all projects, but I just wanted to include some shared, unpublished code from outside of the Firebase functions folder. It looks like @SamChou19815 has a linked commit that's similar, but basically using esbuild you can bundle your entryway file like this:

"scripts": {
    "build": "esbuild src/index.ts --bundle --sourcemap --platform=node --tsconfig=tsconfig.json --external:firebase-* --external:@google-* --outfile=lib/index.js",
}

You need to specify which dependencies not to bundle — I'm starting with anything that starts with firebase- or @google-* although you likely want to specify any published dependencies. I'm not sure if there's a simpler approach vs. specifying all external dependencies, but this approach works well with no additional configuration for my project.

Also took some inspiration from @jackywxd's blog post using Webpack — https://jackywu.ca/2021/2021-01-02/

R-Iqbal commented 2 years ago

There's a lot of solutions here and the most logical seems to be having a monorepo with webpack to bundle up all the content into a deployment folder.

How do these solutions fair when developing with local emulators? If you make a change to a shared package will a manual re-build work be required for the emulators to notice the change?

katopz commented 2 years ago

There's a lot of solutions here and the most logical seems to be having a monorepo with webpack to bundle up all the content into a deployment folder.

How do these solutions fair when developing with local emulators? If you make a change to a shared package will a manual re-build work be required for the emulators to notice the change?

I do have an example (for reproduce this mono-repo issue) that actually working 100% (including re-build) as expected in local but NOT WORKING when deploy here 👉 https://github.com/katopz/firebase-yarn-workspace

Not sure this could be mix with other solutions or not. I already gave up btw.

R-Iqbal commented 2 years ago

There's a lot of solutions here and the most logical seems to be having a monorepo with webpack to bundle up all the content into a deployment folder. How do these solutions fair when developing with local emulators? If you make a change to a shared package will a manual re-build work be required for the emulators to notice the change?

I do have an example (for reproduce this mono-repo issue) that actually working 100% (including re-build) as expected in local but NOT WORKING when deploy here 👉 https://github.com/katopz/firebase-yarn-workspace

Not sure this could be mix with other solutions or not. I already gave up btw.

Oh fun. I'll check out your solution, I wish we had a clear answer which supports sharing local packages with auto re-build during emulator time and support for function deployment.

creativecreatorormaybenot commented 2 years ago

One solution that works for us pretty much without problems at this point:

Use npm workspaces
Add all local packages to workspaces (glob in workspaces entry in top-level package.json)
Write a custom script that npm packs each local package
Add all packed tarballs to your functions directory that you are going to deploy
Add each tar archive as a file: dependency to the package.json of the Cloud Functions directory you are going to deploy
firebase deploy --only functions
Revert steps 3 to 5

0x80 commented 2 years ago

@creativecreatorormaybenot Interesting approach. Are you packing/deploying typescript code?

creativecreatorormaybenot commented 2 years ago

@0x80 yes, exclusively.

This means that you do not need to add any of the local packages as dependencies during development because that is handled by workspaces (you only need to run tsc as you would normally also need to do). When deploying, you go through steps 3 to 7 (because the file: dependencies are required to make firebase deploy work).

R-Iqbal commented 2 years ago

There's a lot of solutions here and the most logical seems to be having a monorepo with webpack to bundle up all the content into a deployment folder. How do these solutions fair when developing with local emulators? If you make a change to a shared package will a manual re-build work be required for the emulators to notice the change?

I do have an example (for reproduce this mono-repo issue) that actually working 100% (including re-build) as expected in local but NOT WORKING when deploy here 👉 https://github.com/katopz/firebase-yarn-workspace

Not sure this could be mix with other solutions or not. I already gave up btw.

Our solutions are really similar. I'm just trying to achieve sharing a local package using yarn workspaces.

I've added the local package @claimsgate/core to my package.json and ran yarn install, the IntelliSense appears to be working correctly so I'm happy that the import is correct.

When I npx tsc and firebase emulators:start --only 'functions' I'm getting Error: Cannot find module '@claimsgate/core', just very unsure if this is an issue with .tsconfig, firebase functions or yarn workspaces

If anyone has any ideas what the root of this issue could be that'd be great! 🗡️

R-Iqbal commented 2 years ago

There's a lot of solutions here and the most logical seems to be having a monorepo with webpack to bundle up all the content into a deployment folder. How do these solutions fair when developing with local emulators? If you make a change to a shared package will a manual re-build work be required for the emulators to notice the change?

I do have an example (for reproduce this mono-repo issue) that actually working 100% (including re-build) as expected in local but NOT WORKING when deploy here 👉 https://github.com/katopz/firebase-yarn-workspace

Not sure this could be mix with other solutions or not. I already gave up btw.

Combine your solution with firelink listed above and you have yourself a solution which automatically re-builds when local changes are made and supports deployments to functions.

firelink deploy --only functions:testFunction performs the copy magic to meet the requirements of all code being inside of the functions folder.

While working locally, let yarn workspaces do the magic.

My package.json is in the functions folder. Appears it should work like a charm!

  "fireDependencies": {
    "@claimsgate/core": "../core" // Firelink will do some magic here when you deploy the function!
  },
  "dependencies": {
    "@claimsgate/core": "1.0.0"
  },
  "fireConfig": {
    "runner": "firebase",
    "outFolderName": ".packages",
    "outFolderLocation": ".",
    "excludes": [
      "node_modules"
    ]
  }

katopz commented 2 years ago

firelink deploy --only functions

Nice to hear! And I confirm it work with firelink in this branch 👉 https://github.com/katopz/firebase-yarn-workspace/tree/firelink

I will keep main branch reference, really hope this can be support directly via firebase-tools.

johanneskares commented 2 years ago

One solution that works for us pretty much without problems at this point:

Use npm workspaces

Add all local packages to workspaces (glob in workspaces entry in top-level package.json)

Write a custom script that npm packs each local package

Add all packed tarballs to your functions directory that you are going to deploy

Add each tar archive as a file: dependency to the package.json of the Cloud Functions directory you are going to deploy

firebase deploy --only functions

Revert steps 3 to 5

This is really one of the best solutions I feel. I'm using yarn workspaces. For reference, here's my firebase.json. Postdeploy is optional if it's running on a CI pipeline.

File Structure:

|-- package.json
|-- packages
|   |-- shared
|   |   |-- package.json
|-- apps
|   |-- firebase
|   |   |-- firebase.json
|   |   |-- functions
|   |   |   |-- package.json

firebase.json

{
  "functions": {
    "predeploy": [
      "cd ../../packages/shared && yarn pack --filename ../../apps/firebase/functions/shared.tgz",
      "cd functions && yarn add ./shared.tgz"
    ],
    "postdeploy": [
      "cd functions && yarn add my-package-name@^1.0.0",
      "rm functions/shared.tgz"
    ],
    "source": "functions"
  }
}

BookmanAG commented 2 years ago

Hey @johnnyoshika I do the same but since I moved to npm workspaces (was using yarn before) I get the error that my package-lock.json file is not in sync with my package.json file, since I updated it with the local dependencies

Here is the error while I am running firebase deploy --only functions:
Build failed: npm ERR! cipm can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.
npm ERR! 
npm ERR! 
npm ERR! Invalid: lock file's @mono-repo/shared@1.3.4 does not satisfy @mono-repo/shared@file:./src/local-dependencies/shared
npm ERR! Invalid: lock file's @mono-repo/shared-config@1.3.4 does not satisfy @mono-repo/shared-config@file:./src/local-dependencies/shared-config
npm ERR! 
I also need to mention that I have 3 packages in my monorepo shared, shared-config and shared-admin. Also shared-admin has a dependency on shared and shared-config.

Before the deploy starts I copy all three packages to my functions/src/local-dependencies. This worked with yarn and yarn.lock but is not working with npm and package-lock.json

Any idea how to resolve this problem? Any hint would be awesome.

I am suffering from the same issue since today as I have been deploying CF with NX Workspaces using @simondotm nx-firebase library to Firebase using Node 10 until today, when for some unknown reason the deployments fail with the same error right after the dist/app-name folder was uploaded successfully.

Anyone has idea of what is happening?

gestureleft commented 2 years ago

Hey @johnnyoshika I do the same but since I moved to npm workspaces (was using yarn before) I get the error that my package-lock.json file is not in sync with my package.json file, since I updated it with the local dependencies Here is the error while I am running firebase deploy --only functions:
Build failed: npm ERR! cipm can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.
npm ERR! 
npm ERR! 
npm ERR! Invalid: lock file's @mono-repo/shared@1.3.4 does not satisfy @mono-repo/shared@file:./src/local-dependencies/shared
npm ERR! Invalid: lock file's @mono-repo/shared-config@1.3.4 does not satisfy @mono-repo/shared-config@file:./src/local-dependencies/shared-config
npm ERR! 
I also need to mention that I have 3 packages in my monorepo shared, shared-config and shared-admin. Also shared-admin has a dependency on shared and shared-config. Before the deploy starts I copy all three packages to my functions/src/local-dependencies. This worked with yarn and yarn.lock but is not working with npm and package-lock.json Any idea how to resolve this problem? Any hint would be awesome.
I am suffering from the same issue since today as I have been deploying CF with NX Workspaces using @simondotm nx-firebase library to Firebase using Node 10 until today, when for some unknown reason the deployments fail with the same error right after the dist/app-name folder was uploaded successfully.

Anyone has idea of what is happening?

I was having the same issue. This comment solved the issue for me: https://github.com/firebase/firebase-tools/issues/653#issuecomment-859613782

jim-alexander commented 1 year ago

{
  "functions": {
    "predeploy": [
      "cd ../../packages/shared && yarn pack --filename ../../apps/firebase/functions/shared.tgz",
      "cd functions && yarn add ./shared.tgz"
    ],
    "postdeploy": [
      "cd functions && yarn add my-package-name@^1.0.0",
      "rm functions/shared.tgz"
    ],
    "source": "functions"
  }
}

This was looking like a good solution until I ran into a caching issue when trying to update my dependancy.

when doing yarn add ./shared.tgz yarn will just decide to use the old version.

I'm assuming because of the version number not changing.

more info: https://github.com/yarnpkg/yarn/issues/6811

sbuys commented 1 year ago

Hey @johnnyoshika I do the same but since I moved to npm workspaces (was using yarn before) I get the error that my package-lock.json file is not in sync with my package.json file, since I updated it with the local dependencies Here is the error while I am running firebase deploy --only functions:
Build failed: npm ERR! cipm can only install packages when your package.json and package-lock.json or npm-shrinkwrap.json are in sync. Please update your lock file with `npm install` before continuing.
npm ERR! 
npm ERR! 
npm ERR! Invalid: lock file's @mono-repo/shared@1.3.4 does not satisfy @mono-repo/shared@file:./src/local-dependencies/shared
npm ERR! Invalid: lock file's @mono-repo/shared-config@1.3.4 does not satisfy @mono-repo/shared-config@file:./src/local-dependencies/shared-config
npm ERR! 
I also need to mention that I have 3 packages in my monorepo shared, shared-config and shared-admin. Also shared-admin has a dependency on shared and shared-config. Before the deploy starts I copy all three packages to my functions/src/local-dependencies. This worked with yarn and yarn.lock but is not working with npm and package-lock.json Any idea how to resolve this problem? Any hint would be awesome.
I am suffering from the same issue since today as I have been deploying CF with NX Workspaces using @simondotm nx-firebase library to Firebase using Node 10 until today, when for some unknown reason the deployments fail with the same error right after the dist/app-name folder was uploaded successfully. Anyone has idea of what is happening?
I was having the same issue. This comment solved the issue for me: #653 (comment)

I'm having the same issue. Have not touched any config and I'm getting this error after 1 year of no issues: Invalid: lock file's @mono-repo/utils@0.0.1 does not satisfy @mono-repo/utils@file:libs/utils

jjgriff93 commented 1 year ago

Using npm workspaces with a /common library shared across an Angular app and Firebase functions, works really well from the Angular side but running into the same woes when deploying the functions:

Build failed: npm ERR! code E404
npm ERR! 404 Not Found - GET https://registry.npmjs.org/@company/common - Not found
npm ERR! 404 
npm ERR! 404  '@company/common@*' is not in this registry.

I find it very frustrating that we're not able to build functions in our own CI environment and have to go through Cloud Build - causes numerous issues like this and both Azure and AWS offer much better solutions to this

johnnyoshika commented 1 year ago

@jjgriff93 Have you tried this technique I described above? I have a very similar setup as you (using npm workspaces) and the technique's been working well for me for over a year.

jjgriff93 commented 1 year ago

@johnnyoshika I did a similar technique and had some luck with it thank you - however in the end I decided to add a step in my deployment pipeline to publish my common library to Google Artifacts registry, then when Firebase deploys them it uses my .npmrc file to find my common package from the registry. So locally I'm able to use symlinks as normal with nom workspaces, then when I deploy it uses the published package

johnnyoshika commented 1 year ago

@jjgriff93 An interesting approach. Thanks for sharing!

GuillaumeSD commented 1 year ago

For anyone still having trouble with this, I use a workaround that imho is better than those detailed here.

For instance with @johnnyoshika technique, your common package's dependencies won't be taken into account, which means you might run into some errors if it has dependencies that your functions package.json doesn't have. In addition, manually editing your package.json file (with sed in this case) is not a safe solution imo. @johnnyoshika feel free to correct me if I misunderstood your technique.

I use NPM Workspaces for my monorepo setup with TypeScript and Gitlab CI. Here's how I got it to work :

Build dependent packages (like a common package) using TypeScript's tsc.
Pack those dependent packages using npm pack to the functions folder you want to deploy that uses those packages (you will get a .tgz file for each dependent packages) => npm pack -w common -pack-destination package-to-deploy/functions
Delete or rename (temporarily, just for the next command to work) the root package.json file, its the one where workspaces are listed => rm package.json or mv package.json package-temp.json
Install the dependent packages from their .tgz file using npm install in the functions folder => npm i common-sdk-1.0.0.tgz
firebase deploy --only functions

Here are the advantages of this technique :

You don't need any additional packages or plugins
All your dependent packages's dependencies will be taken into account at deployment
You don't manually edit any package.json file

sbuys commented 1 year ago

Hello - I am currently out of office on vacation. I will return your message when I return on Monday, 8/29.

If urgent, please call me directly.

0x80 commented 1 year ago

@GuillaumeSD When using your approach during development, do the packages in your workspace pick up changes in code from their dependencies, or do you have to run tsc and npm pack again for changes to propagate?

willviles commented 1 year ago

I made willviles/firebase-yarn-workspaces to make this issue less annoying for my use case.

I use Turborepo as a monorepo build pipeline on top of yarn workspaces. Here's a simple little shell script to deploy locally, which:

Prunes & builds your Turborepo into a clean /out dir
Installs dependencies
Copies & links packages via relative file path replacement in package.json
Deploys all in one go

#!/bin/sh
cd $PATH_TO_TURBOREPO_ROOT

echo 'Creating a pruned Turborepo'
yarn turbo prune --scope=$FUNCTIONS_PACKAGE_NAME

echo 'Copying Firebase config'
cp ./{.firebaserc,firebase.json} ./out

cd ./out

echo 'Installing dependencies'
yarn

echo 'Running Turbo build'
yarn turbo run build --scope=$FUNCTIONS_PACKAGE_NAME --include-dependencies --no-deps --no-cache

echo 'Running firebase-yarn-workspaces'
npx firebase-yarn-workspaces --scope=$FUNCTIONS_PACKAGE_NAME

echo 'Deploying'
firebase deploy --force --only functions

echo 'Cleaning up ./out folder'
rm -Rf ./out

Is it perfect? No. Does it work? Yes!

GuillaumeSD commented 1 year ago

@0x80 The approach I explained is to be used in the deploy stage.

If you meant about auto completion when you import a function from your common package in another package, VS code does that automatically.

If you want your code changes to be propagated during development, you should :

add in your tsconfig.json : "references": [{"path": "./path-to-common"}]
add the -b options when you build your package and the --watch options too if needed This will automatically rebuild your package if code changes happens.

evelant commented 1 year ago

4.5 years later and we still have to use hacks to build firebase functions in a monorepo. Monorepos are incredibly common now even in the smallest of projects. A little love from google on the dev tooling front would be appreciated!

segevfiner commented 1 year ago

We are using a pnpm monorepo and hitting a failure due to npm failing to parse workspace: deps, but we are bundling our functions code so we don't actually need any node_modules... I wonder if there is any sane way to workaround this... e.g. Run pnpm pack via predeploy and have firebase used the packed code instead...

willviles commented 1 year ago

@segevfiner I built firebase-pnpm-workspaces for this purpose!

shinaBR2 commented 1 year ago

I tried many ways and finally get success with a hacky way, hopefully, Firebase CLI will support it soon. Detailed in my blog.

Solution: https://github.com/Madvinking/pnpm-isolate-workspace

The idea is, build all the dependencies into one single workspace with some tweaks in the package.json file since firebase deploy command does not support the pnpm workspace:* protocol.

In my case, the api (apps/api folder) is the name of the workspace I am working with the Firebase Cloud Functions, which has a dependency is the core (packages/core folder) workspace.

The folder structure should be like

root
  |- apps
    |  api
  |- packages
    |  core
  firebase.json
  pnpm-workspace.yaml

Steps:

Change the firebase.json source field to "source": "apps/api/_isolated_", and remove the predeploy hook
Build Cloud Functions locally, the output will be in apps/api/dist
Run pnpx pnpm-isolate-workspace api at the root folder, it will create the folder name _isolated_.
Copy the build folder into new created folder cp -r apps/api/dist apps/api/_isolated_
Go to the apps/api/_isolated_ run mv package.json package-dev.json
Go to the apps/api/_isolated_ run mv package-prod.json package.json
Go to the apps/api/_isolated_ run sed -i 's/"core\"\: \"workspace:\*\"/"core\"\: \"file\:workspaces\/packages\/core\"/g' package.json, thanks to this comment
Finally, run firebase deploy --only functions at the root folder

Example working github action: https://github.com/shinaBR2/shinabr2-world/actions/runs/3770430938/jobs/6410259517

0x80 commented 1 year ago

@shinaBR2 Seems like this is inspired by the Vercel monorepo solution which I am also tinkering with at the moment. So thanks for the write-up, this is very helpful!

Maybe you can explain the motivation behind some of the steps. For example:

Since isolating a workspace creates a self-contained workspace, I wonder why you don't build from there, instead of copying the build folder into the isolated workspace?
What's the reason behind renaming the package files?

shinaBR2 commented 1 year ago

Sure @0x80

For your first question, I will test it later, I did it during rush time 😮‍💨 but I think it should work, will update the result soon.
The second thing, I figured out during running "firebase deploy" command, even I built things successfully but as long as the package.json file contains something like <package_name>: "workspace:*" it will throw the error code EUNSUPPORTEDPROTOCOL so that's why I need to do the dirty part using sed

firebase / firebase-tools

Support mono-repos in deployment #653

Version info

Steps to reproduce

Expected behavior

Actual behavior

Predeploy phase

Postdeploy phase