Feature Proposal: Support monorepo structures where external dependencies are not bundled with webpack (add an example)

[florianbepunkt]

This is a (Bug Report / Feature Proposal)

Feature Proposal

Description (short)

It would be great to get an example for typescript monorepos, using lerna, yarn workspaces or npm workspaces, where node_modules are not bundled with webpack.

Description (detail)

Basically serverless-webpack works fine in a monorepo if you include all dependencies. But in most serverless projects I did, sooner or later I encountered some webpack-incompatible modules. Using nodeExternals however does not work in a monorepo.

If you exclude them in webpack.config.js with nodeExternals like so

externals: [
    nodeExternals({}),
    nodeExternals({
      modulesDir: path.resolve(__dirname, "../../node_modules"),
    }),
  ]

you'll get a runtime error

{
    "errorType": "Runtime.ImportModuleError",
    "errorMessage": "Error: Cannot find module '@nighttrax/bar'\nRequire stack:\n- /var/task/src/index.js\n- /var/runtime/UserFunction.js\n- /var/runtime/index.js",
    // stack omitted for brevity
}

After this I tried to set the node_modules path to the root in the serverless webpack config (which is new in 5.5):

custom:
  webpack:
    webpackConfig: "./webpack.config.js"
    keepOutputDirectory: true
    packager: "npm"
    includeModules:
      nodeModulesRelativeDir: "../../"

This leads to a webpack compile time error: npm ERR! 404 Not Found - GET https://registry.npmjs.org/@nighttrax%2fbar - Not found

Not sure why, but I believe this is due to the way serverless-webpack handles dependencies. If I understand correctly, the code is bundled with webpack, saved to the .webpack folder along with a copy of package.json and then you run an npm install in this folder. Is this correct? If so, I guess this is where things fall apart in my example.

For feature proposals: Using a monorepo has benefits for some projects. It would be great to have an example / support for monorepos with serverless-webpack. However if you have webpack incompatible deps this does not work (see above)

Similar or dependent issue(s):

• 825

• Issue in a great example monorepo that supports lerna, npm workspaces, yarn workspaces, etc. https://github.com/NiGhTTraX/ts-monorepo/issues/50

Example repo

This is an example using npm7 workspaces: https://github.com/florianbepunkt/ts-monorepo/tree/npm

Additional Data

• Serverless-Webpack Version you're using: 5.5.0
• Webpack version you're using: 5.37.0
• Serverless Framework Version you're using: 2.41.2
• Operating System: Mac OSX 10.15
• Stack Trace (if available):

[pevers]

Hi @florianbepunkt ,

I tried to come up with an example to have Serverless + Webpack + Typescript in a Monorepo setup using Yarn workspaces. I used Lerna to keep it ergonomic (deploy from the root folder, run tests, etc.). I is also possible to exclude dependencies from being bundled, I demonstrated that by integrating with Prisma. Repo is here: https://github.com/pevers/swat

If this is heading in the right direction I can open a PR if you want @j0k3r ?

[florianbepunkt]

@pevers This looks great. I would advocate for Yarn2 (or a Yarn2 branch) as the hoisting config has changed and they moved away from noHoist.

[pevers]

@pevers This looks great. I would advocate for Yarn2 (or a Yarn2 branch) as the hoisting config has changed and they moved away from noHoist.

Thanks! I tried upgrading to yarn2, but I run into some incompatibilities. I had to downgrade Typescript, but even then it would fail because serverless-webpack is doing a yarn install --non-interactive --frozen-lockfile. And --non-interactive seems to be deprecated in yarn2.

EDIT: I see you ran into the same issue ;) https://github.com/serverless-heaven/serverless-webpack/issues/642

[cullylarson]

This worked for me, using yarn. It assumes:

1. You're building a project in app/api, relative to your project root.
2. Your serverless.yml file is in app/api/serverless.yml.
3. You're using a monorepo, with workspaces defined in your root package.json.
4. Your app/api/package.json doesn't need to define workspaces because it itself is a workspace within the monorepo.

Steps

1. Edit app/api/package.json and add workspaces for each of the workspaces that app/api depends on:

"workspaces": [
  "../../../../packages/*",
],

These paths are relative to app/api/.webpack/dependencies, which is where the build's package.json file will end up.

This will be ignored by yarn during dev, but will be used during build (see later steps).

2. Create an empty file named yarn.lock and put it in app/api/build-files. This will be copied into your build folder and it will tell yarn that this is the "root" of the project. Without it, yarn will traverse back down to your actual project root and will fail yarn install because the build's folder isn't set up as a workspace.

You can't use your project's root yarn.lock because yarn install will fail with it too.

3. Add the following to your serverless.yml:

webpack:
  packager: "yarn"
  includeModules:
    nodeModulesRelativeDir: ../../
  packagerOptions:
    # This is an empty lock file. It tells yarn to not consider the build
    # folder as part of the monorepo. Without it, `yarn install` will fail.
    lockFile: "build-files/yarn.lock"
    noFrozenLockfile: true
    copyPackageSectionNames: ["dependencies", "workspaces"]

nodeModulesRelativeDir is the path to your root project's node_modules.

lockFile is set to the empty yarn.lock you created in the previous step.

noFrozenLockfile. This needs to be true because your empty yarn.lock will be modified.

copyPackageSectionNames. This tells serverless-webpack to copy the workspaces prop from app/api/package.json into the build folder's package.json.

That's it.