Sam build duplicates dependency packages even if they share the same manifest

[kalpaitch]

Description:

This appears to be closely related to https://github.com/aws/aws-sam-cli/issues/1072, however, this issue has been closed citing "de-duplicated builds, which will build same CodeUri for multiple functions".

The issue I'm encountering specifically focuses on de-duplicating dependency packages.

sam build is creating the dependency folders multiple times, even though all the lambdas in my template use the same manifest file (in this case a package.json) and therefore should be able to share the packaged dependency.

For a template with 10+ functions and large dependency package sizes this can use up a substantial amount of disk space which can be a problem for CI runners.

As far as I can see there isn't an option for sharing dependencies within sam build

Steps to reproduce:

Lambda function core details:

  Lambda1Function:
    Type: AWS::Serverless::Function
    Runtime: nodejs18.x
    Metadata:
      BuildMethod: esbuild

Project structure summary:

project
+-- lambda_1
|   +-- lambda_handler.ts
+-- lambda_2
|   +-- lambda_handler.ts
+-- package.json
+-- template.yml

Command: sam build --cached

Observed result:

.aws-sam
+-- build
+-- deps
|   +-- 0f280905-35ff-4b37-98c2-233b766710d2
|    |   +-- node_modules
|   +-- 40c1b906-e0b8-4762-83a1-03a9c32a15a0
|    |   +-- node_modules
|   +-- d28e775d-8805-4847-913c-47b7cefb2648
|    |   +-- node_modules
+-- build.toml

Expected result:

In this example I would expect only one set of dependencies within .aws-sam/deps/ because all the functions share the same manifest file.

Additional environment details

SAM CLI, version 1.108.0 Typescript project with esbuild

[lucashuy]

Hi, the build dependencies are cached based on the manifest file (the package.json file), and the value of the CodeUri for each of the functions. If you tried moving the handler files to the same directory (and updating the corresponding CodeUris in the template), you should see that the deps are shared. Alternatively, you can utilize a Lambda Layer to share common dependencies, instead of having a single package.json shared between Lambda Functions.

[kalpaitch]

I'll address those two suggestions separately:

1. a) Changing the directory structure is a far from ideal solution in our use case. b) Nor do I feel caching the dependencies should be related to the CodeUri, I'm referring to the .aws-sam/deps/ directory, which only contains npm dependencies, not the .aws-sam/build/ or /.aws-sam/cache directories. c) many of these lambdas are in the same directory, and with the CodeUri specified it's still building to different .aws-sam/deps/ directories, see the manifest hash and CodeUri below are all the same but the dependency folder is different:

   Manifest file is changed (new hash: ea718884c53ee6b0028d7c088f8fefeb) or dependency folder (.aws-sam/deps/70501117-f47d-4d63-84e2-1ed000911263) is missing for (Lambda1Function)                                                                                     
   Building codeuri: backend/api/src/handlers/subfolder1
   Manifest file is changed (new hash: ea718884c53ee6b0028d7c088f8fefeb) or dependency folder (.aws-sam/deps/5af21d52-0418-420a-82bd-775da3082ff6) is missing for (Lambda2Function)                                                                           
   Building codeuri: backend/api/src/handlers/subfolder1
   Manifest file is changed (new hash: ea718884c53ee6b0028d7c088f8fefeb) or dependency folder (.aws-sam/deps/1c6738a0-e3f2-4f74-9c6e-f983a96cd008) is missing for (Lambda3Function)                                                                              
   Building codeuri: backend/api/src/handlers/subfolder1

2. We are using esbuild to bundle these functions which is a far better solution in this case than use a layer, so this isn't an option.

In short, I think there probably is still an issue here, either documentation related or build related.

[KWLandry-acoustic]

I've got the same issue, my template defines a layer and three lambda functions, which results in 4 .aws-sam/deps/ folders of exactly the same content, and I get the same manifest file change messaging, and, as noted, multiple .aws-sam/deps/ folders all with exactly the same node_modules content. If it's inconsequential to the Lambdas deployment in that they all use the same layer and it is a single layer not a collection of 4 deps folders then all is fine but I don't think that's the case.