search

nrwl / nx

Smart Monorepos · Fast CI
https://nx.dev
MIT License
23.52k stars 2.34k forks source link

Allow development of Firebase Cloud Functions #836

Closed Bielik20 closed 3 years ago

Bielik20 commented 6 years ago

Cloud Functions are a great way of developing a micro-service based backend. Just like with node backend it is common to share types and some business logic between the two. Could we come up with some way of integrating them with an Nx Workspace? What are the things that people would like to see in that area?

Current Behavior

No easy way of integrating a Cloud Functions within an Nx Workspace.

Expected Behavior

Developers should be able to easily develop a Cloud Functions backend within an Nx Workspace.

jmarbutt commented 5 years ago

I am interested in this but I am not sure what all I would want on this to make it work easier.

jmarbutt commented 5 years ago

I would like to see it easier to build out the functions project while referencing libraries. Here are some other ideas:

I think that is it off the top of my head.

Bielik20 commented 5 years ago

I think we could divide it into milestones:

  1. MVP
    1. Scaffold Cloud Functions project that uses the root package.json (does not create a new one)
    2. Cloud Functions can reference 'libs'
    3. Triggers
  2. Emulate Firestore / Local development
  3. Unit Testing Functions

Can we agree on that? Did I miss something?

jmarbutt commented 5 years ago

Yeah I think that is perfect

Bielik20 commented 5 years ago

@FrozenPandaz I think this feature has gained quite a popularity. I do realize that there may be other, more important tasks on hand, but we would like to get an update if there are even plans to take it on.

Bielik20 commented 5 years ago

I am not sure it is the right place to post it but... I have made a project with working firebase functions in nx workspace. It uses node-app as a base. You can find it here and here. I think it covers most of the milestones, apart from scaffolding obviously.

nkwood commented 5 years ago

If anyone is using firebase in a nx managed monorepo, you should be aware of this behavior: https://github.com/firebase/firebase-js-sdk/issues/1696#issuecomment-501546596

jongood01 commented 5 years ago

@Bielik20 Setting "source": "/" in your firebase.json is not ideal at all as 'firebase deploy' will then try to upload all your source code. I've been banging my head against a wall try to get this to work. Ideally 'source' would be something like 'dist/apps/functions'.

Bielik20 commented 5 years ago

@jongood01 Yeah I agree but then firebase would complain that there is no npm package there:

Error: No npm package found in functions source directory. Please run 'npm init' inside dist/apps/functions

It is annoying but I think there is little harm in keeping source code up there. It cannot be accessed by anyone from the outside. Deployment is a little longer but also nothing one should worry about.

Or is there a specific reason against it you have in mind?

jongood01 commented 5 years ago

@Bielik20 No other reason but on our team we run the deploy as part of a CI process and the mono repo is going to get huge with a lot of code that has nothing to do with the code on Firebase so builds will get slower and slower. I had a hack around to create a minimal package.json in the dist folder but this was also a hack around and not ideal either. Will pick one or the other option for now. Not really the place for this but I'm really struggling to get Universal rendering working with this as well. I like your idea of having a lib to handle the express server part that gets imported into the cloud function but requiring the main server bundle throws errors.

spy4x commented 5 years ago

Am I right - this issue affects deployment of Nest.js app as well? Thanks Nx team for all the efforts! Nx is awesome and I really appreciate your work :)

Btw, any movements on this issue?

johannesnormannjensen commented 5 years ago

Hi @Bielik20 and @jongood01

I had also run into the problem of not wanting to upload all my source code. What i found is that you can configure functions to ignore certain directories in firebase.json like this: 

...
"functions": {
      "predeploy": [
        "npm run lint functions",
        "npm run build functions --prod"
      ],
      "ignore": [
        "**/apps/web/**",
        "**/apps/web-e2e/**",
        "firebase.json",
        "**/.*",
        "**/node_modules/**"
      ],
      "source": "/"
    },
...

Documentation here: https://firebase.google.com/docs/hosting/full-config

george43g commented 4 years ago

If nrwl could work natively with firebase cloud functions, that would be ideal.

I'm building an app with Firebase. The amount of hours I've spent fixing configuration errors and trying to get local emulation and testing to work is far higher than time spent doing actual programming. Nothing works, ever. I've already tried Lerna but it seems to be more-so designed for open source projects. Not to mention that its package hoisting breaks everything, meaning I need to npm install each package manually anyway, defeating the purpose.

I was hoping that nrwl would finally help in this regard. A setup that would allow me to start actually programming again rather than adjusting config files all day.

spy4x commented 4 years ago

Guys, I've found a workaround, that allows using single package.json for Nx monorepo with Firebase Functions.

TLDR: https://github.com/spy4x/nx-with-firebase-functions

Idea:

Firebase Functions need only 2 files to work:

  1. index.js (you could name it differently, like main.js) with your exported Functions
  2. package.json with next minimal template:
{
  "main": 'index.js',
  "dependencies": { ...your production dependencies here } 
}

So we can use @nrwl/node or @nrwl/nest or other node-based application, build it using $ nx build <appName> (bundle will appear in dist/apps/<appName>) and generate a new package.json file inside dist/apps/<appName> with dependencies that only needed for our application (for example, avoiding Angular/React application dependencies).

I created a simple JS script that gets a list of appName-specific dependencies from the package.json section that I called firebase-functions-dependencies.

And it works fine!

You can run $ firebase emulators:start and test app locally.

The only flaw of this workaround that I see now - it's necessary to manually update root package.json file's firebase-functions-dependencies section when you add a new dependency for your Firebase Functions application.

Check my demo repo (link above) for details. I split changes into commits, so you can see that there is no magic behind. The main change is in this commit: https://github.com/spy4x/nx-with-firebase-functions/commit/c95997976df1f985c2fce146708cb26ace3f5208

I hope that helps somebody. And maybe someone will find a way to update firebase-functions-dependencies automatically.

P.S. you can use any triggers, dependencies and frameworks (ie Nest.js) with this approach. Just don't forget to update firebase-functions-dependencies.

JoelCode commented 4 years ago

Google Cloud Functions Generator

Generate a Google Cloud Function within a Nx workspace with dev tools:

I took Nx development strategy for front-end components & applied it to back-end microservices (Google Cloud Functions). With my plugin, I can create & deploy production-ready microservices in 5 minutes. I then combine my microservices to develop a business automation strategy, business analytics, or data streaming pipeline.

GitHub Repository

FrozenPandaz commented 4 years ago

@spy4x That's amazing! Thank you for sharing!

We have started to provide the ability for the community to provide plugins for Nx and I think this would be a great candidate. If @spy4x or somebody else would be interested in maintaining a plugin for Nx + Firebase functions integrations, we would be happy to help and promote it!

jaytavares commented 4 years ago

My take on @spy4x's solution above. I added the depcheck package as a dev dependency and use it to automate the discovery of unused dependencies within the functions project and then remove those from the generated package.json file.

This means that there's no more need for a separate firebase-functions-dependencies section in your package.json file! 🎉 

const packageJson = require('../../package.json') // Take root package.json
const path = require('path')
const fs = require('fs').promises
const depcheck = require('depcheck')

const ROOT_PATH = path.resolve(__dirname + '/../..')
const distProjectPath = `${ROOT_PATH}/dist/apps/functions`

console.log('Creating cloud functions package.json file...')

let packageJsonStub = {
  engines: { node: '10' },
  main: 'main.js',
}

depcheck(
  distProjectPath,
  {
    package: {
      dependencies: packageJson.dependencies,
    },
  },
  unused => {
    let dependencies = packageJson.dependencies
    if (unused.dependencies.length > 0)
      console.log('Deleting unused dependencies:')
    unused.dependencies.reduce((acc, dep, i) => {
      console.log(`${i + 1} - Deleting ${dep}`)
      delete acc[dep]
      return acc
    }, dependencies)

    fs.mkdir(path.dirname(distProjectPath), { recursive: true }).then(() => {
      fs.writeFile(
        `${distProjectPath}/package.json`,
        JSON.stringify({
          ...packageJsonStub,
          dependencies,
        })
      )
        .then(() =>
          console.log(`${distProjectPath}/package.json written successfully.`)
        )
        .catch(e => console.error(e))
    })
  }
)
vdjurdjevic commented 4 years ago

@jaytavares Great job! I have one more improvement for this setup. Instead of custom scripts in package.json, we could add builders with @nrwl/workspace:run-commands. That way it feels more integrated with nx, and also enables usage of affected --target=command. For example we could add command for generating package.json, edit serve command to run emulator, and add deploy command.

{
    "pkgJson": {
          "builder": "@nrwl/workspace:run-commands",
      "options": {
        "command": "ts-node scripts/build-firebase-package-json.ts",
        "cwd": "tools"
      }
    },
    "serve": {
    "builder": "@nrwl/workspace:run-commands",
    "options": {
        "command": "firebase emulators:start --only functions --inspect-functions"
    }
    },
    "shell": {
    "builder": "@nrwl/workspace:run-commands",
    "options": {
        "command": "firebase functions:shell --inspect-functions"
    }
    },
    "deploy": {
    "builder": "@nrwl/workspace:run-commands",
    "options": {
        "command": "firebase deploy"
    }
    }
}
jaytavares commented 4 years ago

@vdjurdjevic 👍 That's the same approach I used to be able to use nx. Though, I renamed and wrapped the node builder in order to just tack the tool script run at the end of nx build.

"architect": {
  "build-node": {
    "builder": "@nrwl/node:build",
    "options": { },
    "configurations": { }
  },
  "build": {
    "builder": "@nrwl/workspace:run-commands",
    "options": {
      "commands": [
        {
          "command": "nx run functions:build-node"
        },
        {
          "command": "node tools/scripts/build-cloud-functions-package-file.js"
        }
      ],
      "parallel": false
    },
nartc commented 4 years ago

@vdjurdjevic @jaytavares Thanks for the snippets. I was able to get Firebase Functions up and running with NestJS within a Nx workspace. However, I need to run build first before I can even serve with firebase emulators and there's no live reload at all (aka watch). Were you guys able to get --watch working for firebase emulators?

PS: Not sure if it's worth mentioning but initially I have source: "/" in firebase.json but that didn't work because firebase emulators keeps saying: Cannot find module /path/to/my_workspace_name

vdjurdjevic commented 4 years ago

Watch works. Just run nx build functions(or your project name) --watch, and in the separate terminal run emulators. You should have live reload. At least works with my setup. I used node project instead of nextjs, but that should not be an issue.

vdjurdjevic commented 4 years ago

I opened an issue in firebase repo to support a more flexible project structure for functions. That would enable better integration with Nx, and I would be willing to contribute full-featured plugin for firebase (not just functions, but firestore config, hosting, storage, etc..). @FrozenPandaz can you please check it out and tell me what you think?

dalepo commented 4 years ago

I created an angular workspace and then I wrapped my functions under a node app but I'm struggling to use imported libraries since I'm getting this error when trying to deploy firebase functions:

Error: Cannot find module '@nrwl/workspace' at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15) at Function.Module._load (internal/modules/cjs/loader.js:562:25) at Module.require (internal/modules/cjs/loader.js:692:17) at require (internal/modules/cjs/helpers.js:25:18) at Object. (/workspace/decorate-angular-cli.js:28:20) at Module._compile (internal/modules/cjs/loader.js:778:30) at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10) at Module.load (internal/modules/cjs/loader.js:653:32) at tryModuleLoad (internal/modules/cjs/loader.js:593:12) at Function.Module._load (internal/modules/cjs/loader.js:585:3)

Anyone faced this? Any help is appreciated.

ildemartinez commented 4 years ago

Hello

I have the same issue.... any news about that?

imagen

dalepo commented 4 years ago

I found out that this is related to the postinstall script with the angular-cli decorator (decorate-angular-cli.js)

One quick way to make it work is removing the postinstall script from package.json before deploying to functions, then add it back again so the nx client doesn't break. Is there a way to skip the postinstall script when deploying to firebase functions?

beeman commented 4 years ago

I found out that this is related to the postinstall script with the angular-cli decorator (decorate-angular-cli.js)

One quick way to make it work is removing the postinstall script from package.json before deploying to functions, then add it back again so the nx client doesn't break. Is there a way to skip the postinstall script when deploying to firebase functions?

You're probably looking for the --ignore-scripts command:

The --ignore-scripts argument will cause npm to not execute any scripts defined in the package.json. Source here.

dalepo commented 4 years ago

@beeman thanks, I tried it but it doesn't work on the firebase cli

jhlabs commented 4 years ago

@spy4x I have been using your solution and it works great for one function (in my case SSR rendered NextJS). However, how are you deploying several functions with the firebase command? As far as I could find, you can only define one source directory in the firebase.json. If I set this to e.g. the dist folder, firebase will complain again that there is no valid package.json.

spy4x commented 4 years ago

@jhlabs

  1. I export all my cloud functions in one main.ts file, which is then compiled down to dist/<your_path>/main.js. This way all my functions are exported and visible to Firebase.

  2. To give firebase a valid package.json I use custom script that is running during predeploy stage.

jhlabs commented 4 years ago

@spy4x thanks for the response. I have two separate apps, a NextJS SSR app and a NestJS API backend. Each of them should be deployed as separate cloud functions. I could not figure out how to do it with your (better) version of having separate package.json files, but am using the one from the main project for now. In order for Firebase to find the two functions I need to:

  1. For each predeploy of the two apps I need to copy a main.js file into their directory, so dist/apps/nextjs/main.js and dist/apps/api/main.js.
  2. Then I need to also create an index.js file for Firebase to import those two separate directories in the dist/apps directory.

It works for now, but there is definitely room for improvement 😆

spy4x commented 4 years ago

@jhlabs I would recommend to have 2 libs that export NextJS and NestJS apps and 1 app called "cloud functions" that imports those 2 libs and sets up 2 cloud functions for each lib.

Something like this:

import { nextJSApp } from '@yourcompany/next-js-app'; // library #1
import { nestJSApp } from '@yourcompany/nest-js-app'; // library #2

export const nextApi = functions.https.onRequest(async (req, res) => {
  return nextJSApp.handle(req, res);
});

export const nestApi = functions.https.onRequest(async (req, res) => {
  return nestJSApp.handle(req, res);
});

This way you can export both APIs, but keep them separate.

jhlabs commented 4 years ago

@spy4x thanks, this will work well for my use case! The downside that is inherent with this is if you have many cloud functions at some point you cannot deploy them individually right? So the nx affected command would then still pull in that cloud function library and redeploy a function that might not have changed.

dobromyslov commented 4 years ago

@jaytavares your great script does not work with NodeJS 8. And NX by default installs types for Node 8. I rewrote the script in TypeScript and upgraded @types/node to 12.19.0 in the main package.json. And this Node upgrade is required for Google Functions since Google has ditched the Node 8 recently and pushed everyone to migrate to NodeJS 10.

dobromyslov commented 4 years ago

Combined solutions from @spy4x @jaytavares @vdjurdjevic and added my own pipes and tape along with original Firebase config. Works rather well.

workspace.json or angular.json (depends on your initial NX workspace creation options):

Expand

```json { ... "projects": { ... "functions": { "root": "apps/functions", "sourceRoot": "apps/functions/src", "projectType": "application", "prefix": "functions", "schematics": {}, "architect": { "build-node": { "builder": "@nrwl/node:build", "options": { "outputPath": "dist/apps/functions", "main": "apps/functions/src/main.ts", "tsConfig": "apps/functions/tsconfig.app.json", "assets": ["apps/functions/src/assets"] }, "configurations": { "production": { "optimization": true, "extractLicenses": true, "inspect": false, "fileReplacements": [ { "replace": "apps/functions/src/environments/environment.ts", "with": "apps/functions/src/environments/environment.prod.ts" } ] } } }, "build": { "builder": "@nrwl/workspace:run-commands", "options": { "commands": [ { "command": "nx run functions:build-node" }, { "command": "copyfiles .runtimeconfig.json dist/apps/functions" }, { "command": "ts-node tools/scripts/build-firebase-functions-package-json.ts functions" }, { "command": "cd dist/apps/functions && npm install --package-lock-only" } ], "parallel": false }, "configurations": { "production": { "prod": true } } }, "serve": { "builder": "@nrwl/workspace:run-commands", "options": { "command": "nx run functions:build && firebase emulators:start --only functions --inspect-functions" } }, "shell": { "builder": "@nrwl/workspace:run-commands", "options": { "command": "nx run functions:build && firebase functions:shell --inspect-functions" } }, "start": { "builder": "@nrwl/workspace:run-commands", "options": { "command": "nx run functions:shell" } }, "deploy": { "builder": "@nrwl/workspace:run-commands", "options": { "command": "firebase deploy --only functions" } }, "logs": { "builder": "@nrwl/workspace:run-commands", "options": { "command": "firebase funcions:log" } }, "lint": { "builder": "@nrwl/linter:eslint", "options": { "lintFilePatterns": ["apps/functions/**/*.ts"] } }, "test": { "builder": "@nrwl/jest:jest", "options": { "jestConfig": "apps/functions/jest.config.js", "passWithNoTests": true } } } } } ... } ```

tools/scripts/build-firebase-functions-package-json.ts:

Expand

```typescript import * as path from 'path'; import * as depcheck from 'depcheck'; import * as fs from 'fs'; async function main(): Promise { const args = process.argv.slice(2); if (!args?.length || !args[0]) { throw new Error('Application name must be provided.'); } const APPLICATION_NAME = args[0]; console.log(`Application name: ${APPLICATION_NAME}`); /***************************************************************************** * package.json * - Filter unused dependencies. * - Write custom package.json to the dist directory. ****************************************************************************/ const ROOT_PATH = path.resolve(__dirname + '/../..'); const DIST_PROJECT_PATH = `${ROOT_PATH}/dist/apps/${APPLICATION_NAME}`; const PACKAGE_JSON_TEMPLATE = { engines: { node: '12' }, main: 'main.js', }; const packageJson = require('../../package.json'); console.log('Creating cloud functions package.json file...'); // Get unused dependencies const { dependencies: unusedDependencies } = await depcheck( DIST_PROJECT_PATH, { package: { dependencies: packageJson.dependencies, }, } ); // Filter dependencies const requiredDependencies = Object.entries( packageJson.dependencies as {[key: string]: string} )?.filter(([key, value]) => !unusedDependencies?.includes(key)) ?.reduce<{[key: string]: string}>(( previousValue, [key, value] ) => { previousValue[key] = value; return previousValue; }, {}); console.log(`Unused dependencies count: ${unusedDependencies?.length}`); console.log(`Required dependencies count: ${Object.values(requiredDependencies)?.length}`); // Write custom package.json to the dist directory await fs.promises.mkdir(path.dirname(DIST_PROJECT_PATH), { recursive: true }); await fs.promises.writeFile( `${DIST_PROJECT_PATH}/package.json`, JSON.stringify({ ...PACKAGE_JSON_TEMPLATE, dependencies: requiredDependencies, }, undefined, 2) ); console.log(`Written successfully: ${DIST_PROJECT_PATH}/package.json`); } main().then(() => { // Nothing to do }).catch(error => { console.error(error); }); ``` ### firebase.json ```json { "hosting": [ { "target": "console", "public": "dist/apps/console", "ignore": [ "**/.*" ], "headers": [ { "source": "*.[0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f][0-9a-f].+(css|js)", "headers": [ { "key": "Cache-Control", "value": "public,max-age=31536000,immutable" } ] } ], "rewrites": [ { "source": "**", "destination": "/index.html" } ] } ], "functions": { "source": "dist/apps/functions" }, "firestore": { "rules": "firestore.rules", "indexes": "firestore.indexes.json" }, "storage": { "rules": "storage.rules" }, "emulators": { "functions": { "port": 5001 }, "firestore": { "port": 8080 }, "hosting": { "port": 5000 }, "pubsub": { "port": 8085 }, "ui": { "enabled": true } } } ```

package.json dependencies:

Expand

```json "dependencies": { "firebase-admin": "^9.4.2", "firebase-functions": "^3.13.1", }, "devDependencies": { ... "@types/node": "~12.12.38", "copyfiles": "^2.4.1", "depcheck": "^1.2.0", "firebase-functions-test": "^0.2.3", "firebase-tools": "^9.2.1", "ts-node": "~9.1.1", ... } ```

apps/functions/src/main.ts :

Expand

```typescript import * as functions from 'firebase-functions'; export const helloWorld = functions.https.onRequest((request, response) => { response.send('Hello from Firebase!'); }); ```

pascalbe-dev commented 3 years ago

Has someone already started building an nx plugin?

KingDarBoja commented 3 years ago

I want to add to @dobromyslov answer that his approach works with local emulator as well but needs an extra step at the build commands, as the emulator requires .runtimeconfig.json to load environment variables for cloud functions (AFAIK).

The .runtimeconfig.json file must be copied into the dist folder to make it work, like below: copyfiles -E .runtimeconfig.json dist/apps/your-app-api-name

So ends up being:

"commands": [
  {
     "command": "nx run functions:build-node && copyfiles -E .runtimeconfig.json dist/apps/functions"
  },
  {
     "command": "ts-node tools/scripts/build-firebase-functions-package-json.ts"
  }
],
dobromyslov commented 3 years ago

@KingDarBoja thank you!

markwhitfeld commented 3 years ago

Just to add an additional tweak to @dobromyslov's awesome comment:

The angular.json "build" configuration needs the configurations section in order to support the --prod flag and to pass it on to the "build-node" configuration.

Note: I was also having an issue where it was saying 'ts-node' is not recognized as an internal or external command (even though it was installed as a dependency), so I also added the npx call to the command that uses this.

        "build": {
          "builder": "@nrwl/workspace:run-commands",
          "options": {
            "commands": [
              {
                "command": "nx run functions:build-node"
              },
              {
                "command": "npx ts-node tools/scripts/build-firebase-functions-package-json.ts"
              }
            ],
            "parallel": false
          },
          "configurations": {
            "production": {
              "prod": true
            }
          }
        }

Specifying the --prod flag will allow for a more optimised build of the function and to support the production environment configuration that was supplied in the example.

EthanSK commented 3 years ago

There is this community plugin, not sure how well it works: https://github.com/JoelCode/gcp-function

dobromyslov commented 3 years ago

@markwhitfeld thank you for the addition. I also did that but did not post here.

@EthanSK

There is this community plugin, not sure how well it works: https://github.com/JoelCode/gcp-function

gcp-function plugin does not provide full flexibility as such as you develop a standalone Firebase project. Standard Firebase approach lets you mix multiple functions as well as mix various triggers (e.g. firestore document change, storage file write, scheduler) and create a full featured data processing pipeline.

dobromyslov commented 3 years ago

Does anybody know how to properly create and manage 2 or more Firebase projects under one NX workspace?

I am struggling one drawback of @spy4x @jaytavares @vdjurdjevic approach: can't split dependencies between functions. I added one function which requires puppetteer package. As you may know this dependency is rather heavy and takes much time during functions deployment.

My idea concludes in dependencies separation between functions or even better between several Firebase projects:

dobromyslov commented 3 years ago

Found a way to create, run local emulators and deploy two or more NX apps with Firebase support using firebase --config firebase.second-app.json (this feature was added in July, 2020 https://github.com/firebase/firebase-tools/issues/1115)

Just add second NX application with Firebase support as described above: https://github.com/nrwl/nx/issues/836#issuecomment-714384504 and add another firebase.second-app.json file to the root folder:

Expand firebase.second-app.json ``` { "functions": { "source": "dist/apps/second-app" }, "emulators": { "functions": { "port": 5011 }, "firestore": { "port": 8091 }, "hosting": { "port": 5010 }, "pubsub": { "port": 8095 }, "ui": { "enabled": true }, "auth": { "port": 9199 } } } ```

And use --config firebase.second-app.json parameter in all your angular.json like this:

"serve": {
  "builder": "@nrwl/workspace:run-commands",
  "options": {
    "command": "nx run second-app:build && env-cmd firebase --config firebase.second-app.json emulators:start --only functions --inspect-functions"
   }
},
johanchouquet commented 3 years ago

Found a way to create, run local emulators and deploy two or more NX apps with Firebase support using firebase --config firebase.second-app.json (this feature was added in July, 2020 firebase/firebase-tools#1115)

Just add second NX application with Firebase support as described above: #836 (comment) and add another firebase.second-app.json file to the root folder:

Expand firebase.second-app.json And use --config firebase.second-app.json parameter in all your angular.json like this:

"serve": {
  "builder": "@nrwl/workspace:run-commands",
  "options": {
    "command": "nx run second-app:build && env-cmd firebase --config firebase.second-app.json emulators:start --only functions --inspect-functions"
   }
},

I think this is the proper way of doing things within a monorepo & cloud functions. Firebase CFs are powerfull, but in monorepos, we have quite a lot of code. Only one node application (i.e. functions) for these very versatile needs is not possible.

So, I see functions as a node application, integrated within Nx workspace, with an angular.json file. It gives a lot of flexibility to develop / test / deploy the right CFs for the right project we have in the monorepo, which uses a specific Firebase environment. The ability to specify which firebase.jsonfile to take into account is a game changer. I'll try in the near future to do just what i described.

What are your best pratices guys ?

IainAdamsLabs commented 3 years ago

Now that NX node build supports generatePackageJson (https://nx.dev/latest/angular/node/build#generatepackagejson) does this negate the need for the separate build-firebase-functions-package-json script?

jaytavares commented 3 years ago

Now that NX node build supports generatePackageJson (https://nx.dev/latest/angular/node/build#generatepackagejson) does this negate the need for the separate build-firebase-functions-package-json script?

Yes, that's what it means for me at least. I was just able to test the generatePackageJson functionality and it works great. My setup used my script above which used the depcheck package in order to remove unneeded dependencies from the resulting package.json file. I also had renamed my "build" builder to "build-node" and called it from another "build" builder which ran my custom script afterward. The new setting obviates all that and works great. The package.json file is generated and includes only the dependencies that you actually import in your functions code. I was able to add the additional elements needed for cloud functions projects by adding the following package.json file to my functions application:

{
  "engines": {
    "node": "12"
  },
  "main": "main.js"
}

Big 👍 to the Nx team for their implementation of the generatePackageJson feature.

dobromyslov commented 3 years ago

@IainAdamsLabs thank you for the hint. Seems like this tool simplifies boilerplate. But it still lacks another feature - generate package-lock.json. This is optional but it increases deployment speed of Google Cloud Functions. To do it I run additional command for each Cloud Functions project in the workspace like this: cd dist/apps/functions && npm install --package-lock-only and then deploy to the cloud.

shelooks16 commented 3 years ago

Am I the one here who doesn't see the provided solution with webpack good enough?

Firebase Functions processed with webpack potentially have bad outcome. With only one function it's good, but with more functions the webpack solution doesn't seem pretty sweet. At least the provided solution doesn't fit my use case.

I rolled up a fully custom solution which uses just typescript compiler (no webpack) with opportunity to import shared local libraries.

Final functions folder structure for apps/functions/src:

.
└── src
    ├── firestore
    │   ├── profile
    │   │   ├── onCreate.f.ts
    │   │   └── onDelete.f.ts
    │   └── chat
    │       ├── onUpdate.f.ts
    │       └── messages
    │           └── onCreate.f.ts
    ├── pathAliases.ts
    ├── sharedLibs.ts
    └── main.ts

Firestore structure:

.
├── profile
│   └── {uid}
└── chat
    └── {chatId}
        └── messages
            └── {msgId}

Sample firestore/profile/onCreate.f.ts:

import * as functions from 'firebase-functions';
import * as admin from 'firebase-admin';
import { User } from '@WORKSPACE/shared/types';

const firestore = admin.firestore();

export default functions
  .firestore.document('profile/{profileId}')
  .onCreate(async (snapshot, context) => {
    // ...
    // const u: User = { ... };
  }

As you may notice, folder structure reflects Firestore structure. All functions to be deployed are suffixed with .f.ts. They are automatically picked-up and exported from main.ts.

With given profile collection in Firestore, profile/onCreate.f.ts and profile/onDelete.f.ts are functions that run for profile collection documents. The same logic applies to subcollections with given folder/file structure: chat/messages/onCreate.f.ts - the function will run when a document inside messages subcollections is created.

Deployed functions names will reflect camelCased folder structure. Deployed firestore/profile/onCreate.f.ts will be named profileOnCreate, while firestore/chat/messages/onCreate.f.ts => chatMessagesOnCreate

With below files you will be able to achieve that structure with NX + have an ability to import local libs + use tsc compiler instead of webpack.

Source code files

apps/functions/src/main.ts

Expand ```ts import './pathAliases'; import * as admin from 'firebase-admin'; import glob from 'glob'; import camelCase from 'camelcase'; import { resolve } from 'path'; admin.initializeApp(); // add more folders for function detection if needed const lookupFolders = ['firestore']; const baseFoldersRegex = `(${lookupFolders.join('|')})`; const EXTENSION = '.f.js'; const files = glob.sync(`./+${baseFoldersRegex}/**/*${EXTENSION}`, { cwd: __dirname }); for (let f = 0, fl = files.length; f < fl; f++) { const file = files[f]; const functionName = camelCase( file .split(EXTENSION) .join('') .split('/') .join('_') .replace(new RegExp(baseFoldersRegex), '') ); if ( !process.env.FUNCTION_NAME || process.env.FUNCTION_NAME === functionName ) { // eslint-disable-next-line const mod = require(resolve(__dirname, file)); exports[functionName] = mod.default || mod; } } ```

apps/functions/src/pathAliases.ts

Expand ```ts import ModuleAlias from 'module-alias'; import path from 'path'; import { getAliasesForSharedLibs } from './sharedLibs'; // paths in tsconfig.base.ts will let typescript recognize the paths // paths specified and registerd by module-alias will register paths for // compiled javascript const paths = { ...getAliasesForSharedLibs() }; const aliases = Object.entries(paths).reduce((acc, [alias, paths]) => { const aliasCorrected = alias.replace('/*', ''); const pp = paths[0].replace('/*', '/'); const pathCorrected = path .join(__dirname, pp) .replace(/\\/g, '/') .replace('.ts', '.js'); return { ...acc, [aliasCorrected]: pathCorrected }; }, {}); ModuleAlias.addAliases(aliases); ```

apps/functions/src/sharedLibs.ts

Expand ```ts export const sharedLibsConfig = { /** * List of all libs in the monorepo that functions import */ libs: ['shared/types'], /** * Path to a folder in which output all shared libraries * * @note Folder is relative to functions root folder */ outDir: 'libs' }; export function getAliasesForSharedLibs(): Record { const WORKSPACE_NAME = 'PUT_YOUR_WORKSPACE_NAME_HERE'; const { libs, outDir } = sharedLibsConfig; return libs.reduce((result, libName) => { const alias = `@${WORKSPACE_NAME}/${libName}`; const paths = [`../${outDir}/${libName}/src/index.js`]; return { ...result, [alias]: paths }; }, {}); } ```

You have to only adjust sharedLibs.ts to your workspace:

  1. Change WORKSPACE_NAME to yours
  2. Change sharedLibsConfig.libs to list all local libs, and dependencies of libs that functions import. With given libs/shared/types, you have to include shared/types.

apps/functions/tsconfig.app.json

Expand ```json { "extends": "./tsconfig.json", "compilerOptions": { "outDir": "../../dist", "sourceMap": false, "module": "commonjs", "types": ["node"], "strict": true, "removeComments": true, "esModuleInterop": true }, "exclude": ["**/*.spec.ts"], "include": ["**/*.ts"] } ```

Make sure that tsconfig.app.json has outDir set to "../../dist"

Workspace configuration

firebase.json

Expand ```json { "functions": { "source": "dist/apps/functions", "predeploy": ["nx run functions:build"], "runtime": "nodejs10" } } ```

tools/scripts/copy-shared-libs-for-functions.ts

Expand ```ts import * as path from 'path'; import * as fs from 'fs-extra'; async function main() { const args = process.argv.slice(2); if (!args?.length || !args[0]) { throw new Error('Application name must be provided.'); } const APP_NAME = args[0]; const CFG_FILE = 'sharedLibs.js'; // matches (functions/src/sharedLibs.ts) const ROOT_PATH = path.resolve(__dirname + '/../..'); const DIST_FOLDER = `${ROOT_PATH}/dist`; const { sharedLibsConfig } = await import( `${DIST_FOLDER}/apps/${APP_NAME}/src/${CFG_FILE}` ); if (!sharedLibsConfig?.libs || !sharedLibsConfig?.outDir) { throw new Error('Config for copying shared libs has wrong format.'); } const { libs, outDir } = sharedLibsConfig; console.log('Copying shared libraries:', libs); console.log('Total:', libs.length); console.log( 'Destination:', `${DIST_FOLDER}/apps/${APP_NAME}/${sharedLibsConfig.outDir}` ); const promises = libs.map(async (libName: string) => { const src = `${DIST_FOLDER}/libs/${libName}`; const dest = path.join(DIST_FOLDER, `apps/${APP_NAME}`, outDir, libName); await fs.copy(src, dest, { overwrite: true }); }); await Promise.all(promises); } main() .then(() => { // Nothing to do }) .catch((error) => { console.error(error); process.exit(1); }); ```

tools/scripts/build-firebase-functions-package-json.ts

Expand ```ts import * as path from 'path'; import * as depcheck from 'depcheck'; import * as fs from 'fs'; const PACKAGE_JSON_TEMPLATE = { engines: { node: '10' }, main: 'src/main.js' }; async function main(): Promise { const args = process.argv.slice(2); if (!args?.length || !args[0]) { throw new Error('Application name must be provided.'); } const APPLICATION_NAME = args[0]; /***************************************************************************** * package.json * - Filter unused dependencies. * - Write custom package.json to the dist directory. ****************************************************************************/ const ROOT_PATH = path.resolve(__dirname + '/../..'); const DIST_PROJECT_PATH = `${ROOT_PATH}/dist/apps/${APPLICATION_NAME}`; const packageJson = require('../../package.json'); console.log('Creating cloud functions package.json file...'); // Get unused dependencies const { dependencies: unusedDependencies } = await depcheck( DIST_PROJECT_PATH, { package: { dependencies: packageJson.dependencies } } ); // Filter dependencies const requiredDependencies = Object.entries( packageJson.dependencies as { [key: string]: string } ) ?.filter(([key, value]) => !unusedDependencies?.includes(key)) ?.reduce<{ [key: string]: string }>((previousValue, [key, value]) => { previousValue[key] = value; return previousValue; }, {}); console.log( `Required dependencies count: ${ Object.values(requiredDependencies)?.length }` ); // Write custom package.json to the dist directory await fs.promises.mkdir(path.dirname(DIST_PROJECT_PATH), { recursive: true }); await fs.promises.writeFile( `${DIST_PROJECT_PATH}/package.json`, JSON.stringify( { ...PACKAGE_JSON_TEMPLATE, dependencies: requiredDependencies }, undefined, 2 ) ); console.log(`Written successfully: ${DIST_PROJECT_PATH}/package.json`); } main() .then(() => { // Nothing to do }) .catch((error) => { console.error(error); process.exit(1); }); ```

Root package.json

Expand ```json "dependencies": { "camelcase": "^6.2.0", "fs-extra": "^9.1.0", "glob": "^7.1.6", "module-alias": "^2.2.2", // ... }, "devDependencies": { "@types/fs-extra": "^9.0.7", "@types/glob": "^7.1.3", "@types/module-alias": "^2.0.0", "copyfiles": "^2.4.1", "depcheck": "^1.4.0", "onchange": "^7.1.0", "rimraf": "^3.0.2", "ts-node": "~9.1.1", "typescript": "4.0.2", "wait-on": "^5.2.1", // ... }, ```

workspace.json

Expand ```json "functions": { "root": "apps/functions", "sourceRoot": "apps/functions/src", "projectType": "application", "prefix": "functions", "targets": { "compile": { "executor": "@nrwl/workspace:run-commands", "options": { "commands": [ "npx tsc -p apps/functions/tsconfig.app.json", "npx ts-node tools/scripts/copy-shared-libs-for-functions.ts functions", "npx ts-node tools/scripts/build-firebase-functions-package-json.ts functions" ], "parallel": false } }, "build": { "executor": "@nrwl/workspace:run-commands", "options": { "commands": [ "npx rimraf dist/apps/functions", "nx compile functions", "cd dist/apps/functions && npm install --package-lock-only" ], "parallel": false } }, "compile-dev-assets": { "executor": "@nrwl/workspace:run-commands", "options": { "commands": [ "npx onchange -ik libs/**/*.ts -- npx ts-node tools/scripts/copy-shared-libs-for-functions.ts functions", "npx ts-node tools/scripts/build-firebase-functions-package-json.ts functions", "npx copyfiles apps/functions/.runtimeconfig.json dist" ] } }, "serve": { "executor": "@nrwl/workspace:run-commands", "options": { "commands": [ "npx tsc -p apps/functions/tsconfig.app.json -w", "npx wait-on dist/apps/functions && nx compile-dev-assets functions", "npx wait-on dist/apps/functions/package.json && firebase serve --only functions" ] } }, "shell": { "executor": "@nrwl/workspace:run-commands", "options": { "command": "nx build-dev functions && firebase functions:shell --inspect-functions" } }, "download-runtimeconfig": { "executor": "@nrwl/workspace:run-commands", "options": { "commands": [ "firebase functions:config:get > apps/functions/.runtimeconfig.json" ] } }, "logs": { "executor": "@nrwl/workspace:run-commands", "options": { "command": "firebase functions:log" } }, "lint": { "executor": "@nrwl/linter:eslint", "options": { "lintFilePatterns": ["apps/functions/**/*.ts"] } }, "test": { "executor": "@nrwl/jest:jest", "outputs": ["coverage/apps/functions"], "options": { "jestConfig": "apps/functions/jest.config.js", "passWithNoTests": true } } } }, ```

workspace.json for functions became pretty sophisticated. Here is the description of key targets:

How this works

With given workspace structure:

.
├── apps
│   ├── functions
│   └── webapp
└── libs
    └── shared
        └── types

let's assume we have to share types between functions and webapp (i.e. nextjs):

import { User } from '@WORKSPACE/shared/types';

For webapp no need for any action.

For functions to import shared libs, you will have to add them to sharedLibsConfig.libs inside apps/functions/src/sharedLibs.ts:

// sharedLibs.ts
export const sharedLibsConfig = {
  libs: ['shared/types'], // <-- here
  outDir: 'libs'
};

That's it. After that running nx build functions or nx serve functions will compile functions with shared libs :)

If you look inside the dist/apps/functions folder, the output will be:

.
└── functions
    ├── src
    └── libs
        └── shared
            └── types
  1. Libs specified in sharedLibs.ts will be copied over to compiled functions folder under libs.
  2. Because of the module-alias package and pathAliases.ts, compiled javascript will recognize paths correctly during runtime.
  3. Created folder libs in functions matches the name in sharedLibsConfig.outDir. You can basically put any name there.

The power of module-alias and just copying files around is a very strong combo :)

So far this solution has been working great for me. I hope this will be useful for somebody too!

paulbijancoch commented 3 years ago

A "externalDependencies": "none" in the "build"-"architect"-section for the App (in the angular.json) did the trick for me 🚀

dobromyslov commented 3 years ago

@IainAdamsLabs @jaytavares I tested @nrwl/node:build with generatePackageJson with NestJS. Everything builds very well, emulator runs smooth but when deploy a function to the cloud it throws errors caused by absent packages:

Detailed stack trace: Error: Cannot find module 'passport'
Require stack:
- /workspace/node_modules/@nestjs/passport/dist/auth.guard.js
...

Then I compared package.json generated by @nrwl/node:build with generatePackageJson and by build-firebase-functions-package-json.ts and found out that native NRWL generator skips packages required by NestJS.

github-actions[bot] commented 3 years ago

This issue has been automatically marked as stale because it hasn't had any recent activity. It will be closed in 14 days if no further activity occurs. If we missed this issue please reply to keep it active. Thanks for being a part of the Nx community! 🙏