Feature request: Skip `NodejsNpmEsbuildBuilder:NpmUpdate` step when using `--build-in-source` in a js/ts "monorepo"

[cdriesler]

Describe your idea/feature/enhancement

We are currently making great use of the new --build-in-source flag for sharing js/ts code between several AWS resources (also written in js/ts). Our project is set up in a "monorepo" structure, something like:

packages/
  package-a/
    package.json
endpoints/
  endpoint-a/
    package.json
package.json

We use npm workspaces in the root package.json to allow resources like endpoint-a to specify modules like package-a as a dependency. The built-in SAM esbuild options can then find and bundle the code when using --build-in-source.

We've noticed, though, that overall build times have increased by quite a bit because of (what seems to be) a redundant NodejsNpmEsbuildBuilder:NpmUpdate step for each resource. In our case, all of our required dependencies have already been installed before we run sam build. This is true for all resources, because they are all also specified in the workspace's package.json.

The second step of the build for each resource (NodejsNpmEsbuildBuilder:EsbuildBundle) represents less than 5% of the build time. If we could intelligently or explicitly skip the NpmUpdate step, our stack builds would speed up considerably.

Proposal

• Option A: An explicit flag like --skip-dependencies to turn off SAM's management/installation of dependencies. Potentially also requiring --build-in-source. (It feels like that might be the only case where this new flag makes sense.)
• Option B: Skip redundant installs that can be asserted based on the contents of a root package.json. This seems a bit antithetical to the way SAM thinks about resources and handles building, but may be technically possible.

[cdriesler]

Somewhat related: this would also resolve our issue in #6491

[hnnasit]

Hi @cdriesler, thanks for the feature request. I will bring this up with the team and PM's.

[hnnasit]

We've noticed, though, that overall build times have increased by quite a bit because of (what seems to be) a redundant

This is interesting as npm caching should be able to reduce the build times. Would you be able to provide a sample project for us to investigate why build times are increasing?

[RobertMarton1985]

We are in the middle of updating our SAM project and we've ended up with the same project structure. We also use npm workspaces and we also have shared custom dependencies like package-a that is used by our lambda endpoints. Using the --build-in-source flag is mandatory in this case otherwise package-a cannot be resolved when esbuild tries to create the bundle. It runs an npm install but since package-a is not on the registry, it fails.

Anyway, we have found that for each endpoint an npm update is executed with the --install-links flag which makes the build really slow. For example the esbuild bundle command is executed in 1-2 seconds, but the npm update takes more than a minute. And it is executed for each lambda function one-by-one. We have checked the logs during the npm update and it looked like it was running on the ROOT project instead of the lambda. We guess this is because the workspaces do not have a package-lock.json file.

Note: We have noticed that the only way to get rid of the built in @aws-sdk v3 to be deployed with the lambda is to set the `External˙ build properties of the lambdas to an empty array []. With this setting we can achieve around 200ms cold start times. Every other settings resulted in much longer cold starts 600ms+. We strongly believe this is caused by the presence of the default aws sdk. This also prevents us to use layers because those dependencies must be listed in the Externals property otherwise the esbuild failes. But if we add them as externals (eq. the Externals is not an empty array) then the default aws-sdk is also deployed. We definiately want to avoid that.

Note 2: We have a workaround but it is not ideal at all. We do not use the --build-in-source flag and we perform additional tasks via lerna on each lambda before sam build:

1. In each lambda package.json we have a script to bundle the code: esbuild ./ --bundle --platform=node --outfile=bundle.js --minify. We use this script to bundle the code.
2. After the bundle is created we remove the dependencies block completely from the package.json file.

During the sam build the esbuild is still performed but at that point the bundle is already created and it has no additional dependencies so it completes without error.

[adesso-os]

Another AWS product that doesn't understand that there are more package managers in the NodeJS space than npm, and then also doesn't provide any way to work around these annoying issues. Working with any software product out of AWS is such an endless pain.

The real workaround is to drop SAM ASAP and implement a real build.

[RobertMarton1985]

@hnnasit is there any update on this?