Add support for Rush monorepos

[dominicbartl]

Hi @0x80,

first, thanks a lot for all the time you're spending on this issue.

I have a Rush mono-repo (though I think this shouldn't make a difference) and I'm currently replacing my version of bundling dependencies using your firebase-tools-with-isolate.

I have the following directory structure:

The directory packages/apps/backend contains a set of Firebase functions I want to deploy. The content of my firebase.json is the following:

{
"functions": [
  {
    "source": "packages/apps/backend",
    "codebase": "platform-backend",
    "runtime": "nodejs20",
    "ignore": [".rush", "node_modules", "src", ".env*", "*.log", "tsconfig.json", "webpack.config.js"],
    "isolate": true
  }
 ]
}

When running firebase deploy, I get the following error:

It seems to me that it's trying to isolate the root directory instead of the backend directory. It should be looking for a tsconfig.json in packages/apps/backend. Is there anything I'm missing?

I tried putting the isolate.config.json file in the backend directory (same error) as well as in the root directory (throws error since there's no package.json in the root dir)

[hugocxl]

1. How are you building your backend repo? It seems like outDir is missing in your tsconfig.
2. In firebase.json, you might point your source to the folder generated by isolate

Hope that helps

@dominicbartl

[dominicbartl]

@hugocxl Thanks for your reply. I'm using webpack to bundle the source code, though there's still a tsconfig.json in the backend.

Here are the files:

tsconfig.json

{
    "extends": "../../../common/config/tsconfig-node.base.json",
    "compilerOptions": {
        "baseUrl": ".",
        "outDir": "dist"
    }
}

tsconfig-node.base.json

{
  "$schema": "https://json.schemastore.org/tsconfig",
  "display": "Node 16",
  "compilerOptions": {
    "lib": [
      "es2021"
    ],
    "module": "commonjs",
    "target": "es2021",
    "types": ["node", "mocha"],
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "forceConsistentCasingInFileNames": true,
    "experimentalDecorators": true,
    "emitDecoratorMetadata": true,
    "sourceMap": true,
    "resolveJsonModule": true,
    "paths": {
      "@hero/core": ["../../libs/core"]
    }
  }
}

webpack.config.js

const { resolve, join } = require('path');
const nodeExternals = require('webpack-node-externals');
const TerserPlugin = require('terser-webpack-plugin');

module.exports = {
    entry: resolve(__dirname, 'src/index.ts'),
    mode: 'production',
    devtool: 'source-map',
    watchOptions: {
        aggregateTimeout: 200,
    },
    stats: {
        errorDetails: true,
    },
    optimization: {
        minimizer: [
            new TerserPlugin({
                terserOptions: {
                    keep_classnames: true,
                },
            }),
        ],
    },
    module: {
        rules: [
            {
                test: /\.tsx?$/,
                use: 'ts-loader',
                exclude: /node_modules/,
            },
            {
                test: /\.(mjml)$/,
                type: 'asset/resource',
            },
        ],
    },
    target: 'node', // in order to ignore built-in modules like path, fs, etc.
    externalsPresets: { node: true },
    externals: [
        nodeExternals({
            //allowlist: [/^@the-hero-group/],
        }),
    ], // in order to ignore all modules in node_modules folder
    resolve: {
        extensions: ['.ts', '.js'],
        alias: {
            '@the-hero-group/core': join(__dirname, '../../libs/core'),
        },
    },
    output: {
        filename: 'main.bundle.js',
        path: resolve(__dirname, 'dist'),
        library: {
            type: 'this',
        },
    },
};

About your (2): Are you sure about that? I'm using the firebase-tools-with-isolate package and as I understand it, during the deployment the source code gets copied into a tmp directory and isolated there. This directory is then used for upload.

[hugocxl]

mmmm I think you're missing the rootDir in tsconfig, but I am just guessing. Can you try adding "rootDir": "./"? About (2): you're right in that case, sorry.

[dominicbartl]

@hugocxl I tried adding rootDir but it didn't change anything. I think the error must occur beforehand, because the message is Failed to find tsconfig at: <root>/tsconfig.json

So it seems it using the directory of the firebase.json, instead of the directory defined under functions.source

[0x80]

It is not common to have a nested structure for your packages, and I don't think isolate-package supports it. See this section

I would advise you to move apps and libs to the top level if you can. Also, if apps/backend is your only package deploying to firebase, I think there should be no reason to have the firebase files in the root of the monorepo. You could colocate them with the backend code.

[0x80]

For debugging it might be helpful to first use isolate-package without the firebase tools. Then you can just execute npx isolate in the directory that you want to isolate, and get that to produce valid output first.

And in the config set "logLevel" to "debug" so that you get verbose output.

[dominicbartl]

Hi @0x80, thanks for your input. I just wanted to give an update since I've been testing a few configurations.

I took your advice and moved the firebase.json into the backend directory. It's not the only application which gets deployed to Firebase, though it definitely makes sense to have the ability to deploy them separately.

I didn't get isolate to work with the Rush.js monorepo setup. The structure of a Rush.js monorepo is too different from other setups. I guess the main issues, which conflict with isolate are:

• there's no root package.json
• the pnpm-workspace.yaml is not in the workspace root (it's under common/temp/pnpm-workspace.yaml, it's autogenerated)
• the pnpm-lock.yaml is not in the workspace root (also in common/temp/pnpm-lock.yaml)

I assume if the isolate function would take those paths as parameters, it would work with Rush.js as well. But currently it's not possible

[0x80]

Thanks for your feedback. Odd that those files are considered "temporary".

But compatibility seems doable then. I will see what is needed to make those paths configurable and not depend on a root manifest file.

Is there anything specific I could use to detect a Rush monorepo? And would common/temp be used in all Rush setups, or is that just a default that users can change at will?

[dominicbartl]

@0x80 Can be detected when there's a rush.json (docs) in the root of the directory.

The rush.json includes a list of included projects in the mono repo. Based on that configuration, the pnpm-workspace.yaml will be generated automatically.

I was mistaken about the lockfile this is located under the path common/config/rush/pnpm-lock.yaml and is not a temporary file.

And yes those paths are fixed and are used by all Rush setups.

If you want, I can set up a basic Rush.js mono repo for you to test your changes

[0x80]

@dominicbartl Thanks, yes if it's not too much work I think that could be very helpful 👍

[dominicbartl]

Alright, I will set it up with a little explanation tomorrow and post the link here.

[dominicbartl]

@0x80 Here you go: https://github.com/dominicbartl/rush-isolate

I included 2 packages apps/functions and a simple lib in libs/is-even

Just let me know if you have any questions or need help

[0x80]

@dominicbartl For PNPM it seems I have it working in 1.13.0-1. You can install it with @next.

There were a few more hoops to jump through than I imagined, but the good thing is that no additional config is required I think.

For details see: https://github.com/0x80/isolate-package/pull/65

If you have Rush monorepos that are using npm or yarn it would be helpful if you could test those as well.

[0x80]

It has now been published in the latest versions of isolate-package and firebase-tools-with-isolate

[dominicbartl]

@0x80 Amazing work, thanks a lot for your support. I just tested the new version and I encountered one issue, when deploying the functions.

1. Frozen lockfile

 ERR_PNPM_FROZEN_LOCKFILE_WITH_OUTDATED_LOCKFILE  Cannot perform a frozen installation because the version of the lockfile is incompatible with this version of pnpm

Try either:
1. Aligning the version of pnpm that generated the lockfile with the version that installs from it, or
2. Migrating the lockfile so that it is compatible with the newer version of pnpm, or
3. Using "pnpm install --no-frozen-lockfile".
Note that in CI environments, this setting is enabled by default.

I upgraded to the latest version of pnpm (8.15.6). You can reproduce the error using the test repo and the following commands:

cd apps/functions
rushx isolate
cd isolate
CI=true NODE_ENV=production pnpm install

The generated lockfile version from isolate-package is 5.4. When removing the lockfile and running pnpm install in the isolate directory, the lockfile version is 6.0.

[0x80]

@dominicbartl please make a separate issue for this as it seems unrelated to the Rush compatibility.

I'm not sure I understand the problem. Isn't this a version incompatibility between pnpm on your system and pnpm that is used to read your lockfile in the cloud deployment?

Is the error coming from Firebase deploy?

The part that handles the pnpm lockfile is the same for non-Rush monorepos, and my deployments still work, so I'm trying to understand where it comes from.

[dominicbartl]

@0x80 Found the mismatch, was more pnpm than Rush related. The lockfile in the repo was version 5.4 and Rush didn't update the lockfile after updating pnpm. After deleting the lockfile and installing the dependencies again, the lockfile was version 6 and everything worked as expected.

Not sure why running pnpm install using pnpm 8.15.6 and lockfile version 5.4 works, but running CI=true NODE_ENV=production pnpm install throws the ERR_PNPM_FROZEN_LOCKFILE_WITH_OUTDATED_LOCKFILE error.

Rush commands

This is just for reference if someone is using Rush and stumbles over this post

Rush has the ability to add repo-wide commands using autoinstallers. Autoinstallers are packages with dependencies used by tools/scripts which automatically get installed when a command is run.

# This creates the common/autoinstallers/firebase/package.json file
rush init-autoinstaller --name firebase

Add firebase-tools-with-isolate to common/autoinstallers/firebase/package.json and run

rush update-autoinstaller --name firebase

To add the command, add this to common/config/rush/command-line.json

{
  "$schema": "https://developer.microsoft.com/json-schemas/rush/v5/command-line.schema.json",
  "commands": [
    {
      "name": "deploy-firebase",
      "commandKind": "global",
      "summary": "Deploys to firebase",
      "autoinstallerName": "firebase",
      "shellCommand": "cd $RUSH_INVOKED_FOLDER && firebase deploy"
    }
  ],
  "parameters": [
    {
      "parameterKind": "string",
      "argumentName": "ONLY",
      "associatedCommands": [
        "deploy-firebase"
      ],
      "longName": "--only",
      "description": "Limit the services which get deployed"
    },
    {
      "parameterKind": "string",
      "argumentName": "PROJECT",
      "associatedCommands": [
        "deploy-firebase"
      ],
      "longName": "--project",
      "description": "Select the Firebase project",
      "required": true
    }
  ]
}

After that, you can run the following command in any directory containing a firebase.json

# Show help
rush deploy-firebase --help
# Run deployment
rush deploy-firebase --project <project>