Is Lerna the right choice for a serverless mono repo?

[ajhool]

The hoisting in Lerna and Workspaces is useful for development but makes the bundle process more difficult.

The creator of Lerna suggests that it might not be used for these types of serverless repos (issue @lerna/lerna#1061 ) where each microservice is not an npm package. Without hoisting, deployment might be as simple as wrapping the node_modules folder with the dist folder and pulling project-references in using the "prepend" option in tsconfig

[tommedema]

I think it is. Reason is that serverless mono repos typically depend on more non-serverless packages than serverless packages; you re-use these packages across your workspace.

In our case we have 8 generic packages and 2 serverless services. The serverless services utilize the generic packages.

Note that in this repo I've disabled hoisting for all packages prefixed with sls-.

Additionally, I believe Lerna allows you to run commands based on a dependency graph: e.g. lerna run deploy should be runnable from the workspace root to deploy all serverless services in one go.

[ajhool]

Ah, the sls- prefix for nohoist is clever. Good idea! And I like the new packaging solution.

Lerna as a script runner makes sense and hopefully yarn adds that as a native Workspace feature

[tommedema]

Thanks for the feedback.

With the new packaging solution are you talking about https://github.com/tommedema/serverless-mono-example/commit/e0f97a081f35b181d55a2523cb583168e685698d ?

I'm not too happy about it yet since it seems a bit hackish and it's quite slow.

[ajhool]

Yes, that packaging solution.

For production install, I'm looking into the --modules_folder flag, which might remove the need to copy node_modules into the dist folder. Instead, the production node_modules would be installed directly into the dist folder. This might also remove the need to call yarn install again in these lines:

cd ../../
yarn install
cd $cwd

because the development node_modules folder would not be overwritten by yarn install --production.

I'll also always build my artifact on a CI/CD server, so I don't need to worry about cleaning up after/before the build, which should remove some of those steps.

Also, does that code pull in project-references? I saw your issue in learn-a but am not sure how that packaging solution would grab the project-references

[tommedema]

I tried this too but it results in this bug: https://github.com/yarnpkg/yarn/issues/6293

[ajhool]

Yikes. You might be able to get away with simply removing the circular dependency (eg. rm /dist/node_modules/sls-random. That should still remove the need to do multiple yarn installs/cleans and I think things would still run properly. It's unclear from the two package system whether other suprises might come when more than two packages are used.

I'm currently considering using Yarn workspaces / Lerna to build + publish packages to npm or local repo, and then in a separate "deploy" stage (using different directories/server than my dev/build stage) I would do a production install and use the --focus flag to prepare the final distribution package. If the goal of Workspaces/Lerna is to move node_modules around but maintain proper package.json pointers on publish... then it makes sense to aggregate dependencies from the output of the publish step

[tommedema]

Sounds confusing! How would this help a typical setup like in this repo?

I'm not sure if it's a good idea to rm sls-random. Since the local install is bugged it might be better to avoid it completely.

[ajhool]

I was overthinking it. We also have slightly different challenges -- you sound like you need to build + deploy from your development environment, whereas I am building + deploying from a build server. In my case, I am stuck with two yarn installs. You would be able to avoid one of the yarn installs if yarn install --production --modules-folder dist/ worked, which it does't.

The "input" to the build server will always be a git commit. Before yarn install and before tsc -b.

1. yarn install (required for tsc build)
2. tsc -b (required - can't push build result to git)
3. yarn install --production (required)
4. copy node_modules to dist, zip, deploy

None of those steps are avoidable and they should all be compatible with Lerna, Workspaces, and Project References. Step 3 might be avoidable if yarn adds support for multi-stage builds or support for keeping production & dev dependencies in separate folders

https://github.com/yarnpkg/yarn/issues/3254 https://github.com/yarnpkg/yarn/issues/2781

You might want to consider using docker for multi-stage builds. The basic idea is that docker would copy the results of your build into another container, excluding node_modules. Then it would do a production install, package, and deploy. Your development environment wouldn't be touched, aside from the build (which you could also use docker to avoid). It might sound worse than it is or maybe it actually is overkill!

https://codefresh.io/docker-tutorial/node_docker_multistage/ https://engineering.busbud.com/2017/05/15/docker-multi-stage/ https://engineering.busbud.com/2017/05/21/going-further-docker-multi-stage-builds/ https://stackoverflow.com/questions/43747776/copy-with-docker-but-with-exclusion

Instead of publishing the final Docker container to a registry, you would use the final Docker container to generate/deploy your zip file.

[tommedema]

Using docker sounds like going back in time!

We shouldn't have to bother with containerization when using serverless.

To be honest, anyone will use a build server for a staging and production environment. However, it is incredibly handy to be able to deploy from local: any developer can spawn their own little environment from their command line. I think you should also strive for this, in addition to building from CICD.

[ajhool]

Docker would only be used as a platform for building the application code or executing the multi-stage build. The app would be pure serverless.

And I need Docker to deploy from local. My mac can't compile the image processing node package called "sharp" which uses some native C libraries to do the image processing; if I compile on mac but execute in AWS Lambda / linux, the code fails. The code is deployed as pure serverless but Docker provides a consistent local build environment that I can share with others

[ajhool]

Just FYI, Serverless' package command executes its own dev dependency exclusion logic by default in production environments. This might solve your yarn install --production issue. I'm using AWS SAM and my own zip logic so I haven't confirmed that serverless' behavior successfully excludes dev dependencies without running the yarn install --production command, but serverless might already do what you were trying to shoehorn it into doing

https://serverless.com/framework/docs/providers/aws/guide/packaging/

https://github.com/serverless/serverless/blob/master/lib/plugins/package/lib/zipService.js

I believe that Serverless' logic is compatible with Lerna but I am currently running into a zip + symlink issue when using Linux' zip tool and Lerna's symlinks

[tommedema]

Nope, the whole reason for doing this is because their package command is bugged when you use yarn workspaces.

On Mon, Aug 27, 2018, 18:30 ajhool notifications@github.com wrote:

Just FYI, Serverless' package command executes its own dev dependency exclusion logic by default in production environments. This might solve your yarn install --production issue. I'm using AWS SAM and my own zip logic so I haven't confirmed that serverless' behavior successfully excludes dev dependencies without running the yarn install --production command

https://serverless.com/framework/docs/providers/aws/guide/packaging/

https://github.com/serverless/serverless/blob/master/lib/plugins/package/lib/zipService.js

— You are receiving this because you modified the open/close state.

Reply to this email directly, view it on GitHub https://github.com/tommedema/serverless-mono-example/issues/1#issuecomment-416375431, or mute the thread https://github.com/notifications/unsubscribe-auth/AAUQOWCHhdOD_NNXIH7kEck2I4zX6Hpwks5uVGT4gaJpZM4WIDLS .

[tommedema]

@ajhool fyi I discovered a new issue with my package method

even if you specify which files should be packaged in each package:

packages/random/package.json:

  "main": "dist/index",
  "typings": "dist/index",
  "files": [
    "dist"
  ],

When you run yarn install --production with nohoisting, the files node_modules will contain all source files of that package:

Do you have another creative solution on how to deal with this?

[ajhool]

the files node_modules will contain all source files of that package:

I'm not certain that we're dealing with the same problem, but I've spent all day figuring out a yarn install --production issue. Are you saying that when you include packageA in sls-packageB, you see all of the source code from packageA inside the node_modules folder for sls-packageB?

That's the problem I have at the moment.. the production install successfully installs only production dependencies in each package's node_modules folder; however, if projectA depends on projectB, then projectA/node_modules/projectB/node_modules will contain all dependencies, including dev, even though projectA/node_modules and projectB/node_modules have the correct production-only dependencies

Are we talking about the same thing?

[tommedema]

@ajhool yes to your first question. I am now running a cleanup script as a workaround:

https://gist.github.com/tommedema/cd5e41fc22ab6c26b31b7eb234238b56

(bottom few lines)

did you run yarn install --production from the workspace root or from the package level? Accroding to https://github.com/yarnpkg/yarn/issues/6293 you must run it only from the workspace root.

[ajhool]

Yes, I'm running yarn install --production from the workspace root.

Also, I removed Lerna from my project and am using local file dependencies... I found that developing on top of both Workspaces and Lerna was too unstable. Local file dependencies are easy and are just what Lerna is using under the hood.

A long time ago (closed): https://github.com/yarnpkg/yarn/issues/3577

Today (open, uncommented): https://github.com/yarnpkg/yarn/issues/6323

Looks like others are finding this issue, too

[justinwaite]

@tommedema Where do you store this script? Root of the folder? Once it creates artifact.zip, how do you tell serverless deploy to use that zip instead of packaging it up on its own?

[tommedema]

@jdeanwaite see https://github.com/tommedema/serverless-mono-example/commit/e0f97a081f35b181d55a2523cb583168e685698d

[medington]

Wanted to share this with anyone who reads this issue. Rather than use the nohoist option, I opted to use a serverless plugin which automatically traverses the directory structure and correctly creates a deployment package which also properly excludes all development dependencies. Just started using it but it seems to work well so far. It's called "serverless-plugin-monorepo".

Found it from this thread: https://forum.serverless.com/t/using-serverless-with-yarn-workspaces/4560