Support mono-repos in deployment

[laurenzlong]

See: https://github.com/firebase/firebase-functions/issues/172

Version info

3.17.3

Steps to reproduce

Expected behavior

Actual behavior

[0x80]

@shinaBR2 The sed command was not compatible with MacOS. This worked for me: sed -i.bak 's/\"core\"\: \"workspace:\*\"/\"core\"\: \"file\:workspaces\/packages\/core\"/' package.json This puts the original source in package.json.bak. Apparently there is no way to not create a backup file on MacOS, but it can't hurt.

I managed to deploy the functions! 🚀 🥳

I also have a few suggestions for improvements:

IMO the firebase stuff should not be in the root of the monorepo. I have moved firebase.json and .firebaserc to apps/api and changed it to:

{
  "functions": {
    "codebase": "test-monorepo",
    "source": "_isolated_",
    "runtime": "nodejs18",
    "ignore": ["node_modules", ".git", "*.log"]
  },
...

Note the codebase property, this allows you to deploy to one firebase project from multiple different repositories, see this for more info, so in case you will create more functions that do not belong to the api, you can have them in their own independent firebase package.

I think it makes sense if firebase-tools is a dev dependency on the packages that you deploy from, so I've run pnpm add -D firebase-tools@~11.18 on the apps/api directory. Somehow 11.19 is giving a vague "unexpected error". Yesterday I also ran into this on another project...

In order to deploy I run pnpm exec firebase deploy --only functions from the apps/api directory, maybe there's a better way, I'm new to pnpm...

Also, from your instructions it wasn't clear to me that the api/dist folder itself should end up in api/_isolated_/dist.

Eventually, I'd want a script to handle all the manual steps of course, but I'm very happy with how this turns out! I think the pnpm + turborepo setup is what I will adopt for my existing projects.

Thanks again for getting me on the right track with the example of isolated workspaces.

[shinaBR2]

Hi @0x80 thanks for your suggestion, they make sense to me too 👍 love it! I will give it a try then.

My initial intention is to try to solve this problem for the CI server like Github actions. We will get many benefits from the monorepo structure during the development stage but we don't want to run the manual stuff like copy files, isolateing workspaces every time we did changes.

So we will go one by one of your concerns.

I think it makes sense if firebase-tools is a dev dependency on the packages that you deploy from

Yes agree but it's more suitable for your local development since you will run firebase deploy command from your local machine. In the CI server, please note that I use this.

What actually it does and WHY do I need to use that? The answer is the "authentication process". In order to deploy Firebase Cloud Functions, "you" need to have the right permissions. For example in your local machine, you need to run the command firebase login before doing anything, then you need to grant access. The same thing will happen on the CI server, we need to grant the right permissions to the Google Service Account through the GCP_SA_KEY key. In the CI environment, there are no browsers to let you sign in, that's the point. So instead of manually running the command pnpm exec firebase deploy in the CI server, the above w9jds/firebase-action will handle things for you.

so I've run pnpm add -D firebase-tools@~11.18 on the apps/api directory. Somehow 11.19 is giving a vague "unexpected error". Yesterday I also ran into this on another project

I am not sure for this part, maybe you can try to run pnpm add -D with the --filter instead 🤔

Also, from your instructions it wasn't clear to me that the api/dist folder itself should end up in api/_isolated_/dist

The reason why is in the package.json file, the field "main", you can view my file here: https://github.com/shinaBR2/shinabr2-world/blob/main/apps/api/package.json

The "main" will in the package.json file will tell NodeJS: "hey, look for this file to get start running". In my case, the value is dist/index.js so I need to have the dist folder at the same level with the package.json file. Note that we are mention about the isolated code (the apps/api/_isolated_ in my case)

Eventually, I'd want a script to handle all the manual steps of course

Absolutely! Remember that we have part of using sed command to replace the value in the package.json file? This is the most tricky part! Let's say in the future I will have another dependency in another workspace like packages/abc, then I need to do exact same thing again that I current do with packages/core, and it will be annoying. Anyway, we will figure out the better way soon I hope.

Nice day, hope these things make sense and clearer to you, feel free to input 👍

I also update my blog

[alexyork]

Can we get an update from the Firebase team (@taeold) on when this issue will be fixed? The issue is now four years old, monorepo's are very common (and becoming the default choice for many teams), and there are many people waiting for this...

[creativecreatorormaybenot]

The solution I described in https://github.com/firebase/firebase-tools/issues/653#issuecomment-1064268346 has worked for us without any issues for the past year @alexyork. We have a monorepo and use it to deploy to Firebase.

[arthurgubaidullin]

I have a solution for Nx. I wrote a plugin. It can automatically build in only the necessary dependencies. Behind the scenes it use yalc.

Unfortunately, there is no time to write about it in the blog.

Feel free write questions in issues.

[alexyork]

@creativecreatorormaybenot Yeah, I was planning to try the solution outlined in this comment and now I realise that seems to be based on your solution. I can report back if I get that working.

@arthurgubaidullin Thanks for your help, but I don't use Nx and ideally don't want extra packages/plugins to solve this.

My original comment was more aimed to the Firebase team, as these solutions all seem to be workarounds. Ideally, I would like this to work out of the box via the Firebase CLI tools. Seems like that is a reasonable ask, given how common monorepo's are these days. With npm workspaces they are actually an out-of-the-box option, unsupported by Firebase CLI.

[benrandja-akram]

@arthurgubaidullin @alexyork Here is my solution:

• Bundle functions index.js with vite lib mode
• Mark all dependencies as externals except monorepo packages
• Finally don't install custom monorepo packages since firebase will throw an error if they can't install a package from npm registry.

[arthurgubaidullin]

@benrandja-akram Good option. Thank you for sharing!

Nx have a similar bundling mechanism. I haven't tried it.

What interests me, have you measured cold start performance when concatenating code into one file and when including as local dependencies?

[WesleyYue]

@laurenzlong Given that it seems like the Firebase team is not working on this feature, can you pin a Firelink endorsement on the original post? There's 20 different solutions proposed in this thread over 100 comments. It took me 1-2 days to read through them and try them with different failures and firelink was the most seamless solution that worked out of the box. The mentions of Firelink is now buried in the middle of the thread which makes it difficult to discover. Endorsing the package on the OP will solve 90% of the misery caused by this issue.

[benrandja-akram]

@laurenzlong Given that it seems like the Firebase team is not working on this feature, can you pin a Firelink endorsement on the original post? There's 20 different solutions proposed in this thread over 100 comments. It took me 1-2 days to read through them and try them with different failures and firelink was the most seamless solution that worked out of the box. The mentions of Firelink is now buried in the middle of the thread which makes it difficult to discover. Endorsing the package on the OP will solve 90% of the misery caused by this issue.

using vite to bundle https://github.com/my-org-work-upwork/turborepo-firebase

[simondotm]

@benrandja-akram Good option. Thank you for sharing!

Nx have a similar bundling mechanism. I haven't tried it.

What interests me, have you measured cold start performance when concatenating code into one file and when including as local dependencies?

I can only imagine that using bundlers would be optimal for cold-start if they produce one bundle per function, and then perhaps use the new codebase feature of the functions configuration to specify each function. I'm also imagining that a move to es modules would be a necessary optimization also for decent tree shaking. Of course, this all is predicated on how many dependencies and functions a particular project has.

[GuillaumeSD]

@alexyork The implementation I use and have been using for the past year without any issue is indeed based on @creativecreatorormaybenot implementation.

Following your message, I updated mine to be more specific and comprehensive. I hope you will find it useful.

I might make an example repo of a minimal working example with multiple packages, CI/CD,... using npm workspaces if some people would find it useful ?

[ishowta]

It is simple and easy to deploy the entire monorepo.

1. specify the source of functions as root (.) and ignore unnecessary files /firebase.json

   {
     ...
     "functions": {
       "source": ".",
       "ignore": [
         "firebase.json",
         "**/.*",
         "**/node_modules/**",
         "**/packages/@(web|mobile)/**",
         // Any other large file that is not necessary.
       ]
     },
     ...
   }

2. enter the path to the functions entry file in the main field of package.json in the root. /package.json

   {
      ...
     "main": "./packages/functions/dist/index.js",
      ...
   }

That's all. I have deployed stably in this way.

[alexyork]

@GuillaumeSD @creativecreatorormaybenot I have basically done the same as you guys, with a minor variation. I prefer this approach to the others because it does not require any extra tools/packages/helpers.

I could not get npm install working on the generated common-1.0.0.tgz file. I got some error, but the problem could have been me. Anyway, I had a slight workaround for this which means that I did not need to do any renaming of the root package.json as @GuillaumeSD's updated solution does. Instead, I run this in my Cloud Functions folder to set the dependency to the .tgz file: npm pkg set 'dependencies.@my-app/common'='file:./lib/common-1.0.0.tgz'

The npm pkg command is built-in, and lets you surgically read/write directly to the package.json file, so I use it to update the dependency to point at the .tgz file just before deploying, and change it back after deploying.

Here is my solution in full:

firebase.json:

  "functions": {
    "predeploy": ["bash ./deploy-cloud-functions.sh pre-deploy", "npm --prefix \"$RESOURCE_DIR\" run build"],
    "source": "packages/cloud-functions",
    "postdeploy": ["bash ./deploy-cloud-functions.sh post-deploy"]
  },

deploy-cloud-functions.sh

#!/bin/bash

if [ "$1" = "pre-deploy" ]; then
  echo "Cloud Functions pre-deploy script: Packing shared code..."
  mkdir ./packages/cloud-functions/lib
  npm run build --workspace @my-app/common
  npm pack --workspace @my-app/common --pack-destination ./packages/cloud-functions/lib
  cd packages/cloud-functions
  npm pkg set 'dependencies.@my-app/common'='file:./lib/common-1.0.0.tgz'
  cd ../..
  echo "Cloud Functions pre-deploy script: Done!"
  exit
elif [ "$1" = "post-deploy" ]; then
  echo "Cloud Functions post-deploy script: Cleaning up shared code..."
  cd packages/cloud-functions
  rm -rf lib
  npm pkg set 'dependencies.@my-app/common'='1.0.0'
  cd ../..
  echo "Cloud Functions post-deploy script: Done!"
  exit
else
  echo "Not a valid parameter. Must be one of: 'pre-deploy'|'post-deploy'."
  exit
fi

Yes, you can argue manually editing package.json with npm pkg set is a hack, but I might equally argue that renaming the root package.json file to package.json.temp is also a hack. Anyway, it's working, that's the key thing!

I appreciate the help everyone gave here in this thread, so I tried to do the same by posting my full solution.

[GuillaumeSD]

@ishowta The constraint is that all your cloud functions must be in the same package right ?

[GuillaumeSD]

@alexyork Thanks for sharing your solution. It is indeed very close to our implementation. Great finding to use npm pkg set instead of deleting/renaming the root package.json and npm i the common tgz. Unfortunately I can't use this on my end due to some package-lock.json requirements for deployment in my company.

Happy to know that you manage to get it working too 💪

[ishowta]

@GuillaumeSD What does all your cloud functions must be in the same package mean? It is possible to use shared packages from the functions package using the workspace feature of package manager (I use yarn workspaces).

packages/functions/package.json

"dependencies": {
    "@app/admin": "*",
    "@app/core": "*",
    "@app/models": "*",
    "@app/uni-firebase": "*",
    ...

[GuillaumeSD]

@ishowta I meant all your functions must be exported in your functions package one way or the other. Or I could have misunderstood your implementation. Anyway, all that matters is that you found a solution that fits your needs 👍

[ishowta]

@GuillaumeSD Ah, sorry, I understood what you are saying. It certainly seems difficult to achieve that use case naturally. By the way, I think the advantage of deploying the entire monorepo is that it doesn't have to handle with complex dependencies and that changes to node_modules (lockfile, patches, resolution field, etc.) can be used as is. Thank you for your time.

[isomorpheric]

@segevfiner I built firebase-pnpm-workspaces for this purpose!

This is great - thank you so much @willviles .

[BenJackGill]

5 years, over 120 comments, and still no native solution from Firebase?!

Hoping Firebase fixes this soon. Until then, here is my solution.

It's an extension of @alexyork, which in turn was taken from @GuillaumeSD @creativecreatorormaybenot.

It's basically the same, but I think a little easier to reason about. Also you can use any source as your "functions" folder now because we cd "$RESOURCE_DIR" before making changes.

The code below assumes you have a monorepo named monorepo and a package named shared which is version 0.0.1. You will need to update those parts accordingly.

Step 1: Change predeploy and postdeploy of firebase.json as per below:

  "functions": {
    "predeploy": ["bash ./deploy-cloud-functions.sh pre-deploy", "npm --prefix \"$RESOURCE_DIR\" run build"],
    "source": "packages/cloud-functions",
    "postdeploy": ["bash ./deploy-cloud-functions.sh post-deploy"]
  },

Step 2: Save this bash file as deploy-cloud-functions.sh into your firebase functions source folder, normally that would be /functions unless you've changed it.

if [ "$1" = "pre-deploy" ]; then
  cd "$RESOURCE_DIR"
  echo "Cloud Functions pre-deploy script: Packing shared code..."
  rm -rf temp
  mkdir ./temp
  npm run build --workspace @monorepo/shared
  npm pack --workspace @monorepo/shared --pack-destination ./temp
  npm pkg set 'dependencies.@monorepo/shared'='file:./temp/monorepo-shared-0.0.1.tgz'
  echo "Cloud Functions pre-deploy script: Done!"
  exit
elif [ "$1" = "post-deploy" ]; then
  cd "$RESOURCE_DIR"
  echo "Cloud Functions post-deploy script: Cleaning up shared code..."
  npm pkg set 'dependencies.@monorepo/shared'='0.0.1'
  rm -rf temp
  echo "Cloud Functions post-deploy script: Done!"
  exit
else
  echo "Not a valid parameter. Must be one of: 'pre-deploy'|'post-deploy'."
  exit
fi

Step 3: Now you can run firebase deploy --only functions or whatever other command you normally use to deploy to firebase.

[evelant]

The poor tooling and mountains of restrictions involved in using firebase have pushed me toward a comprehensive solution -- switch to https://supabase.com. You get modern tooling and techniques there, responsive developers, plus much greater capability.

[BenJackGill]

I created a much more robust version of my previous code.

This version loops through ALL the "internal packages" listed under internalDependencies of package.json.

Then it creates a tarball for each one, and updates the devDependencies and dependencies references to point to that tarball. After Firebase Functions is finished deploying it reverts everything back again.

Note: this assumes you have jq installed on your machine. On Mac you can use brew install jq to install it. Not sure about windows, just Google that part.

Step 1: Add internalDependencies to package.json. It should list all your internal dependencies that you want to include in your Firebase Functions. For example:

{
  "name": "functions",
  "license": "UNLICENSED",
  "scripts": {
    "lint": "eslint --ext .js,.ts .",
    "build": "tsc",
    "build:watch": "tsc --watch",
    "serve": "npm run build && firebase emulators:start --only functions",
    "shell": "npm run build && firebase functions:shell",
    "start": "npm run shell",
    "deploy": "firebase deploy --only functions",
    "logs": "firebase functions:log"
  },
  "engines": {
    "node": "16"
  },
  "main": "lib/index.js",
  "dependencies": {
    "firebase-admin": "^11.5.0",
    "firebase-functions": "^4.2.0",
  },
  "devDependencies": {
    "@my-monorepo/eslint-config-custom": "1.0.0",
    "@my-monorepo/tsconfig": "1.0.0",
    "@my-monorepo/types": "1.0.0",
    "eslint": "^8.36.0",
    "typescript": "^4.9.0"
  },
  "internalDependencies": {
    "@my-monorepo/eslint-config-custom": "1.0.0",
    "@my-monorepo/tsconfig": "1.0.0",
    "@my-monorepo/types": "1.0.0",
  }
}

Step 2: Save this bash file as deploy-internal-packages.sh into your firebase functions source folder, normally that would be /functions unless you've changed it.

#!/bin/bash

# Get the deployment stage from the first command-line argument
deploymentStage=$1

# Exit if the param was not given
if [ -z "$deploymentStage" ]; then
  echo "Error: Deployment stage not specified."
  exit 1
fi

# Do the pre-deploy abd post-deploy tasks
if [ $deploymentStage = "pre-deploy" ]; then
  echo "Packinging internal dependencies..."

  #Change into the Firebase Functions source folder
  cd "$RESOURCE_DIR"

  # Remove the temp folder if it alread exists
  rm -rf temp

  # Create a new temp folder
  mkdir ./temp

  # Use jq to extract the dependencies object
  dependencies=$(cat package.json | jq -r '.dependencies')

  # Use jq to extract the devDependencies object
  devDependencies=$(cat package.json | jq -r '.devDependencies')

  # Use jq to extract the internalDependencies object
  internalDependencies=$(cat package.json | jq -r '.internalDependencies')

  # Loop through the internalDependencies object
  for package in $(echo $internalDependencies | jq -r 'keys[]'); do
    # Build each dependency (if it has a build script)
    npm run build --workspace $package --if-present
    # Pack each dependency and add it to the temp folder
    npm pack --workspace $package --pack-destination ./temp
    # Update the "dependencies" and "devDependencies" to point to the tarBall
    version=$(echo $internalDependencies | jq -r ".[\"$package\"]")
    tarBallPackageName=$(echo $package | sed 's/@//g' | sed 's/\//-/g')-$version.tgz
    if [[ $(echo $dependencies | jq -r 'has("'$package'")') == "true" ]]; then
      npm pkg set "dependencies.$package"="file:./temp/$tarBallPackageName"
    elif [[ $(echo $devDependencies | jq -r 'has("'$package'")') == "true" ]]; then
      npm pkg set "devDependencies.$package"="file:./temp/$tarBallPackageName"
    fi
  done

  echo "Finished packinging internal packages!"

  exit
elif [ $deploymentStage = "post-deploy" ]; then
  #Change into the Firebase Functions source folder
  cd "$RESOURCE_DIR"

  echo "Cleaning up internal package code..."

  # Remove the temp folder if it alread exists
  rm -rf temp

  # Use jq to extract the dependencies object
  dependencies=$(cat package.json | jq -r '.dependencies')

  # Use jq to extract the devDependencies object
  devDependencies=$(cat package.json | jq -r '.devDependencies')

  # Use jq to extract the internalDependencies object
  internalDependencies=$(cat package.json | jq -r '.internalDependencies')

  # Loop through the internalDependencies object
  for package in $(echo $internalDependencies | jq -r 'keys[]'); do
    # Update the "dependencies" and "devDependencies" to the original version
    version=$(echo $internalDependencies | jq -r ".[\"$package\"]")
    tarBallPackageName=$(echo $package | sed 's/@//g' | sed 's/\//-/g')-$version.tgz
    if [[ $(echo $dependencies | jq -r 'has("'$package'")') == "true" ]]; then
      npm pkg set "dependencies.$package"="$version"
    elif [[ $(echo $devDependencies | jq -r 'has("'$package'")') == "true" ]]; then
      npm pkg set "devDependencies.$package"="$version"
    fi
  done

  echo "Finished cleaning up internal package code!"
  exit
else
  echo "Not a valid parameter. Must be one of: 'pre-deploy'|'post-deploy'."
  exit
fi

Step 3: Add the bash script to the predeploy and postdeploy hooks of firebase.json so that it runs before and after deployment. For example:

  "functions": {
      "predeploy": [
        "bash \"$RESOURCE_DIR\"/deploy-internal-packages.sh pre-deploy",
        "npm --prefix \"$RESOURCE_DIR\" run lint",
        "npm --prefix \"$RESOURCE_DIR\" run build"
      ],
      "postdeploy": [
        "bash \"$RESOURCE_DIR\"/deploy-internal-packages.sh post-deploy"
      ]
  },

Step 4: Now you can run firebase deploy --only functions or whatever other command you normally use to deploy to firebase.

[Stradivario]

@BenJackGill

Pretty much the same thing that Firelink is doing without the tarbal part

Probably it will be a better idea to refactor it to use your approach with the tarbal.

https://github.com/rxdi/firelink

[pdoreau]

I've a something similar but when depolying a next.js app (framework integration), with yarn workspaces and the following structure :

• project root (unique firebase.json)
  • website (nextjs), depends on internal-dep through yarn workspaces
  • internal-dep
  • ...

yarn firebase deploy build a cloud function for SSR pages, with this error : npm ERR! notarget No matching version found for internal-dep@1.0.0

No problem to run locally (e.g serve)

@BenJackGill Would your solution work in my case ? Should it be adapted ?

[BenJackGill]

@pdoreau Sorry I'm not familiar with Yarn or nextjs. I use Vue and NPM. But the script is pretty well commented out so you can adapt to you liking if needed. Sorry I couldn't be more help :/

[0x80]

@benrandja-akram I tried going down your path of using Vite to bundle my Firebase functions, but it failed and the example you shared seems to be incomplete. The api package doesn't depend on a shared monorepo package, and this is I think the key challenge that people are trying to solve in this thread.

I since learned that Vite was originally meant for bundling apps and libraries for use in browsers, and it doesn't yet support targeting Node.js out of the box. (And I couldn't get it to work at all)

I am now pursuing a strategy using tsup which seems to turn out the way I want, but I have yet to solve some things...

[ArturAmpilogov]

Currently Firebase uses "archiver" for custom zip packaging.

Some ideas for Firebase monorepo support:

1. Instead of custom archive logic switch to the standard npm pack

2. Instead of custom ignore field in firebase config use the standard .npmignore

3. NPM config has bundleDependencies field that can bundle packages during npm pack execution for functions. Internal packages can be specified in that array.

   "bundleDependencies": ["@packages/shared"],
   "dependencies": {
     "@packages/shared": "*"
   }

4. Instead of asking developers to manually specify bundleDependencies, it would be nice to automate the process and modify package.json during the deployment. To find out internal dependencies in a monorepo run npm list --json from the functions directory.

   "version": "1.0.0",
   "name": "monorepo",
   "dependencies": {
     "functions": {
       "resolved": "file:../apps/functions",
       "overridden": false,
       "dependencies": {
          "@packages/shared": {
             "version": "1.0.0",
             "resolved": "file:../../packages/shared",          
          }
       }
   }

[0x80]

I worked on what I would consider the first real solution to this problem, and I'm very pleased with the result!

It is basically zero-config with no manual steps, and transparently hooks into the predeploy configuration.

https://github.com/0x80/isolate-package

I have only tested it with PNPM and Yarn v3, but it was designed to be compatible with any package manager.

Please give it a try and let me know if you run into issues. I have taken great care to write understandable and well-structured code so I expect that any tweaks we need to make at this point would be fairly easy.

One thing I would still like to know is, does Firebase tools deploy actually detect and respect a pnpm-lock.yaml file? Deploying with the --debug flag gives so data, and I couldn't find the answer in there.

Maybe @laurenzlong or someone else from the Firebase team could verify?

I have also written an article discussing the issues with the existing workarounds and the solution I created. I hope I didn't miss anything, but please let me know if you think some statement is incorrect.

https://thijs-koerselman.medium.com/deploy-to-firebase-without-the-hacks-e685de39025e

[nickstamas]

@0x80 Thank you. I was wrestling with this issue all day yesterday, but I was just able to deploy using your package. I had to use npx isolate in the predeploy script though, not isolate.

[0x80]

@nickstamas Thanks, and happy to hear you got it working! What package manager are you using, and where is your firebase.json located in relation to the package you are deploying?

[nickstamas]

I'm using yarn berry workspaces. I have /packages/firebase/functions. firebase.json is in the /firebase directory. Looking into it, I realized I added 'isolate-package' as a dependency at the monorepo root. I removed it and yarn added it to packages/firebase and now it finds the binary without npx.

[0x80]

I don't think nested packages are very common. I vaguely remember seeing an issue from turborepo that it doesn't support nesting yet either. Frankly, I haven't even considered such structures when writing the code, so I hope things are compatible.

Instead, I would choose a flat approach and prefix any firebase package with firebase-, so you get /packages/firebase-functions. The name "firebase-functions" might look a little confusing, and this is why I normally namespace all monorepo packages so it is clear on all imports that they are local packages.

[nickstamas]

@0x80 Interesting. I used Firebase for a long time before coming to monorepos, and I've only ever seen functions nested inside the firebase directory, so I didn't think anything of it. I guess it makes sense when not in a monorepo environment, but probably better to break it out in a monorepo. Either way, it's working, so thanks again.

[Stf-F]

Hi @BenJackGill , just wondered if you could point me in the right direction. I am having a few issues with your script. The script is in the functions folder and the firebase.json and package.json are updated as per your instructions. However, when I run the firebase deploy command, the temp folder gets created, I can see the @myapp/mypackage being printed but the npm pack fails. The packages are not copied over and I end up with the following error: Could not install from "temp" as it does not contain a package.json file. When I try the command npm pack --workspace @myapp/mypackage --pack-destination ./temp manually from within the functions folder, the command fails too with this error No workspaces found, but when I try from the root of the monorepo (where the top package.json containing the references to the workspaces is defined) the command works. So everything seems to be pointing to a path hiccup. But not sure where to take it from there. Any help would be appreciated. Thanks

[BenJackGill]

@Stf-F Sorry not sure why it dosen't work for you. You might have better luck with what @0x80 created: https://github.com/firebase/firebase-tools/issues/653#issuecomment-1538200299

[Stf-F]

Thanks @BenJackGill. Unfortunately the whole ESM/CJS debacle makes @0x80's package not a viable solution for me. I have resorted to some manual labour for now and will reconsider my options at a later stage. Thanks

[0x80]

@Stf-F What do you mean with "the whole ESM/CJS debacle"? AFAIK my solution does not care about which output format you use for each package. As long as package manifests adhere to some common rules as described in the readme, I think things should work.

[Stf-F]

Hi @0x80, I will log an issue on your GH rather than adding comments here not directly linked to the original issue.

[0x80]

I can confirm that Firebase deploy is now picking up the PNPM lockfile, but the adapted lockfile from isolate-package doesn't seem to work yet. I've created an issue here, and as a temporary workaround you now have the ability to exclude the lockfile.

This is only an issue for PNPM users I think because the Yarn and NPM lockfiles are deployed unmodified.

[dominicbartl]

I've been struggling with deployment of mono-repos which use Firebase in multiple projects and. I'm currently using Rush with PNPM workspaces.

1. Currently, during deployment, archiver is used to generate an archive in the tmp directory. Instead of archiving, the functions directory can be copied to a tmp directory excluding the ignored files.
2. Read the package.json file to find local dependencies which are outside the functions directory and need to be packaged as well.
3. Copy those dependencies into an e.g. internal_dependencies directory and rewrite the paths in package.json.
4. Continue to read the pacakge.json files of internal dependencies, copying them and rewriting the paths recursively until all local nested dependencies are included.

@taeold I would love to see this integrated in firebase-tools and since this is the top issue I believe this is useful to a lot of other developers as well. I would be happy to provide a PR if you think this something the CLI could support. Let me know what you think. :smile:

[gauthampait]

I am using Firebase Hosting with NextJS which its dynamic frameworks feature. While there is a solution to bundle the cloud functions with webpack, there isn't anything that we can do to make dynamic frameworks work with MonoRepo.

It's been 5 years now ☹️ Can we have firebase-tools support monorepo.

[0x80]

@dominicbartl @gauthampait did you give isolate-package a try? I don't want to keep pushing it here, but it was designed as a generic and convenient solution, and so far I haven't been made aware of any incompatibilities. PNPM workspaces are supported, and that's what I use myself, although the lockfile is not altered correctly yet so it needs to be excluded (I don't consider this a big problem or risk for my project). Yarn and NPM are not a problem in this regard. I think none of the other proposed workarounds tackle this for PNPM yet either.

If you are still using your own scripts, or you fail to deploy from a monorepo, I'm curious to hear how your needs differ.

I recently started using isolate-package to deploy to firebase from two different packages in my monerepo. One is deploying regular 1st gen firebase functions, and the other is deploying 2nd gen functions, where runtime environment configuration is no longer available.

[dominicbartl]

Hi @0x80, I gave isolate-package a try yesterday and integrated it into our newest project, which uses Rush for mono-repo management. I think isolate-package is not compatible with a Rush mono-repo, since the lockfile and the workspace file are not in the root of the repository (or I missed something :sweat_smile:).

common/config/rush/pnpm-lock.yaml
common/temp/pnpm-workspace.yaml

By adding the possibility to configure those paths in the isolate.config.json it will be compatible with Rush as well.

I really like that you use the files property from the package.json. That's what I used in my script as well, since I prefer to include what's needed instead of include everything by default and excluded what's not needed.

The script I wrote does basically the same thing as isolate-package, just not as generic and configurable, but it works for our current setup.

Though, I still think a lot of people will benefit from having firebase-tools support this out of the box. For example, we heavily use the emulators for local development. Since the source property in the firebase.json has to point to the isolate directory and not to the actual source directory, for every change that should be reflected in the emulator, additionally to the build, the isolate command is required to be executed. I guess (p)npm install also needs to be run every time.

Having the isolation only be done during deployment allows us to quickly develop locally by running webpack --watch and having the changes be automatically reflected in the emulator after the build without any extra tooling.

Since you invested quite some time in this issue, I would love to hear your thoughts on the PR

[0x80]

@dominicbartl Thanks for bringing the emulator issue to my attention. I've been lazy with integration testing, so it didn't occur to me yet. I will try to find a solution for it.

I don't think I will have time soon to give valuable feedback on the PR as I'm not familiar with the firebase-tools code and I have lots of other things on my plate at the moment, but I'm curious to see how generic the solution is an how it differs from what I did.

I agree that eventually, it would be best to have a solution integrated into the firebase-tools deploy command. I was hoping I could someday do that by making isolate-package an internal dependency, but as you made apparent, there is still more work to be done to cover all use cases. Possibly, your approach in the PR would be simpler and more generic.

As far as Rush support, I don't see myself spending time on this, but PRs are welcome of course. Rush, on first impression, seems to be a pretty niche tool. I never heard of it, and it makes me wonder how much adoption there is outside of Microsoft and how it stacks up against newer solutions like Turborepo. And because it takes quite an opinionated approach, I wonder how long it will last. I hope I don't offend anyone by saying that. It's just my gut feeling at this point.

[angelcervera]

I expended the latest two days trying to deploy functions that are part of a monorepo with workspaces. I tried with pnpm originally, because it's the one used. After too much time, I tried with npm and no lucky. And now, I find this issue created in 2018!!! I think that the Google Cloud Functions team should take this a little bit seriously. This is not a good experience at all.

[Stradivario]

I expended the latest two days trying to deploy functions that are part of a monorepo with workspaces. I tried with pnpm originally, because it's the one used. After too much time, I tried with npm and no lucky. And now, I find this issue created in 2018!!! I think that the Google Cloud Functions team should take this a little bit seriously. This is not a good experience at all.

Did you give it a try with https://github.com/rxdi/firelink ?

@angelcervera

[angelcervera]

Did you give it a try with https://github.com/rxdi/firelink ?

@Stradivario I tried the one implemented by @0x80 , but it needs Turbo, and I don't want add and migrate the full project (that is in production with CI/CD and all the stuff) only because it.

Now, I can not expend more time in this stuff. But I will give a try in the future.

[Stradivario]

Did you give it a try with https://github.com/rxdi/firelink ?

@Stradivario I tried the one implemented by @0x80 , but it needs Turbo, and I don't want add and migrate the full project (that is in production with CI/CD and all the stuff) only because it.

Now, I can not expend more time in this stuff. But I will give a try in the future.

This is something that it is working as a replacement for firebase function and it is only 8 kb of code so nothing fancy. I have created this package before more than 3 years due to the same problem. There are many people that are using it and it works as expected :)

You need only to write your dependencies in fireDependencies and after that to run firelink deploy like you would do with firebase deploy and everything is mapped automatically so the package present inside the bundle of the lambda.

Cheers!

[angelcervera]

@Stradivario You've convinced me. I will try (but not today).

Thanks!