(assets): EXTREMELY slow bundling with docker.

[alexandervandekleutab]

General Issue

I have a lambda nodejs function that is taking over half an hour to bundle.

The Question

I have seen the other github issues complaining about docker bundling taking 20 seconds up to around 4 minutes. I am experiencing bundling taking several times longer than that.

    const signDocumentsLambda = new lambda.NodejsFunction(
      this,
      generateName('lambda'),
      {
        entry: path.join(__dirname, '../src/index.ts'),
        handler: 'handler',
        functionName: generateName('lambda'),
        bundling: {
          nodeModules: ['pdfkit'],
          forceDockerBundling: true,
          commandHooks: {
            beforeBundling: () => [],
            beforeInstall: () => [],
            afterBundling: copyAssetsAfterBundling,
          },
        },
      }
    )
...
const copyAssetsAfterBundling = (inputDir: string, outputDir: string) => {
  return [
    `cp -rf "${path.join(inputDir, 'src', 'bin')}" ${outputDir}`, // copy binary assets for pdf libraries
    `cp -rf "${path.join(inputDir, 'src', 'render', 'fonts')}" ${outputDir}`, // copy fonts to /fonts
    `cp -rf "${path.join(inputDir, 'src', 'render', 'images')}" ${outputDir}`, // copy images to /images
  ]
}

I need to copy some assets (fonts and images) after bundling, and I need to include pdfkit binaries to I can do operations on PDFs in my Lambda function. This works and functions just fine, but bundling takes FOREVER. I am writing this issue while waiting for cdk deploy.

Here is what my terminal looks like:

[+] Building 880.6s (3/12)                                                                                                                                                                                  
 => [internal] load build definition from Dockerfile                                                                                                                                                   0.0s
 => => transferring dockerfile: 1.13kB                                                                                                                                                                 0.0s
 => [internal] load .dockerignore                                                                                                                                                                      0.0s
 => => transferring context: 2B                                                                                                                                                                        0.0s
 => [internal] load metadata for public.ecr.aws/sam/build-nodejs14.x:latest                                                                                                                            1.1s
 => [1/9] FROM public.ecr.aws/sam/build-nodejs14.x@sha256:e6ea34c50ba2429ebe4abf0794e611f95b4ddd5713ccebe636e94423649a9464                                                                           879.2s
 => => resolve public.ecr.aws/sam/build-nodejs14.x@sha256:e6ea34c50ba2429ebe4abf0794e611f95b4ddd5713ccebe636e94423649a9464                                                                             0.0s
 => => sha256:b69bee44f907c796c3afc971c0012cb3b8a79c8e3349980ecea5f527e4f28c93 81.58kB / 81.58kB                                                                                                       1.3s
 => => sha256:678592bfe0bcd99289d5de44b93b310014551726dbe149c11545364feeb2c6a8 36.35MB / 36.35MB                                                                                                     190.0s
 => => sha256:3967a671955bd41a974d02b2da91740fda7c47fe9bac4a78b485838ca12d96a3 9.25MB / 9.25MB                                                                                                        50.9s
 => => sha256:6eb4ada699d1c4429779ef6af626cd8302d6d864d7b1564ce7995d87f1162c2a 2.42kB / 2.42kB                                                                                                         0.0s
 => => sha256:249c3fbef36b621cf436ef1ce012c3d98a6d7fc63cc503ae3a83885880b84bd5 5.95kB / 5.95kB                                                                                                         0.0s
 => => sha256:e6ea34c50ba2429ebe4abf0794e611f95b4ddd5713ccebe636e94423649a9464 772B / 772B                                                                                                             0.0s
 => => extracting sha256:b69bee44f907c796c3afc971c0012cb3b8a79c8e3349980ecea5f527e4f28c93                                                                                                              0.1s
 => => sha256:4fd754d9c5c7377545dae9dabf9107cfadb7d8934d436b28811c437d24e5e668 270.53MB / 488.63MB                                                                                                   879.2s
 => => sha256:ee3a83f3774459dc74d298775da20b8cf92933fa735b1ce3e3e2be7df4032acb 147B / 147B                                                                                                            51.4s
 => => sha256:e94bd91809b32c46b6353154bb0c7588d75d42ce5ffd0d0b9dfe0031ec06bc78 43.68MB / 43.68MB                                                                                                     260.6s
 => => extracting sha256:678592bfe0bcd99289d5de44b93b310014551726dbe149c11545364feeb2c6a8                                                                                                              2.1s
 => => sha256:834e9e90d91770f6e65fd4050c35d6003dbbe47400da783bdf3aeb17e1b5b7c9 45.69MB / 45.69MB                                                                                                     424.2s
 => => extracting sha256:3967a671955bd41a974d02b2da91740fda7c47fe9bac4a78b485838ca12d96a3                                                                                                              1.1s
 => => sha256:427cb0c5a38f7c6cfb3c38f5e0274da1425c34990ea0dadfe186bf2aa8f6ae89 95.00kB / 95.00kB                                                                                                     262.9s
 => => sha256:a5a29591c452d213153569792d3656d254ed4ab900190df7c89d83ac4024d672 60.67kB / 60.67kB                                                                                                     264.1s

I have no idea why it is taking so long.

CDK CLI Version

1.121

Framework Version

1.144

Node.js Version

14

OS

macOs

Language

Typescript

Language Version

No response

Other information

No response

[alexandervandekleutab]

Note also that I am using forceDockerBundling: true because without it, the deployment hangs on the first "publishing" stage.

[Richardmbs12]

Same here, we also use a docker image to build our functions and the container performs much slower than what it would if I do a normal docker run

[JesseDeReus]

Same issue here CDK CLI Version: 2.41.0 Node.js Version: 14.5.0 OS: Windows 10

[+] Building 1228.7s (3/12)
 => [internal] load build definition from Dockerfile                                                                                                           0.4s
 => => transferring dockerfile: 32B                                                                                                                            0.4s
 => [internal] load .dockerignore                                                                                                                              0.4s
 => => transferring context: 2B                                                                                                                                0.4s
 => [internal] load metadata for public.ecr.aws/sam/build-nodejs14.x:latest                                                                                    1.4s
 => [1/9] FROM public.ecr.aws/sam/build-nodejs14.x@sha256:33984cf50413ccae4ad26776d05c3841abc2d8bb5dfb8ca3332786c6e4cfd804                                  1226.8s
 => => resolve public.ecr.aws/sam/build-nodejs14.x@sha256:33984cf50413ccae4ad26776d05c3841abc2d8bb5dfb8ca3332786c6e4cfd804                                     0.0s
 => => sha256:2bb1f04acd08172bd17a6a4f0ad8ef5a2d43e96b41cc1347a4a796e6ca7e54b0 105.83kB / 105.83kB                                                             1.8s
 => => sha256:e54c2956418b842a4b3cf7d8b07d87ce8ef88c966c90350374fe84127dc491a2 36.46MB / 36.46MB                                                             409.4s
 => => sha256:33984cf50413ccae4ad26776d05c3841abc2d8bb5dfb8ca3332786c6e4cfd804 772B / 772B                                                                     0.0s
 => => sha256:c79e61004c34b06cd411cf53880889e7267c967b4719d0843d56e4115d44aba6 2.22kB / 2.22kB                                                                 0.0s
 => => sha256:580bfa76f9ea3f1b80352373590794e244d30709ec68bd9ab06bb1029ca6c550 5.89kB / 5.89kB                                                                 0.0s
 => => sha256:ffa1311af52a4252b9f81953c122c6363ae4b2a8e64f4a1bd1251a09964ddbf2 71.46MB / 71.46MB                                                             772.0s
 => => sha256:3bca8f99be6608d20a50f1725bbd216c306897ee09e1cdb89ad9b2367e4a004c 9.26MB / 9.26MB                                                               105.6s
 => => sha256:e8f0d08cbd2ec85d7dddc1f479820ede20d395b5c91a5fbbc2d023d04506a1d2 120.59MB / 504.30MB                                                          1226.8s
 => => sha256:8436e78da53c2a21e47a47e673975f997402c9e3363a0ae1ca97e7f0d773df98 44.98MB / 44.98MB                                                             913.7s
 => => sha256:7bba6c4d9a0bb1d84b48d8297260baa1d3494a34874535f8de842e6b0fb8c12a 47.11MB / 47.11MB                                                            1181.6s
 => => extracting sha256:ffa1311af52a4252b9f81953c122c6363ae4b2a8e64f4a1bd1251a09964ddbf2                                                                      9.9s
 => => extracting sha256:2bb1f04acd08172bd17a6a4f0ad8ef5a2d43e96b41cc1347a4a796e6ca7e54b0                                                                      2.0s
 => => extracting sha256:e54c2956418b842a4b3cf7d8b07d87ce8ef88c966c90350374fe84127dc491a2                                                                     10.7s
 => => extracting sha256:3bca8f99be6608d20a50f1725bbd216c306897ee09e1cdb89ad9b2367e4a004c                                                                      7.4s
 => => sha256:c5dd541c984bf7424fa72a82ece5777c6b21aab68d8a51106d59e7b41ee2d64d 96.21kB / 96.21kB                                                             921.8s
 => => sha256:a2675f1d97ff3a473fa0de6b1c5c46ec6cc81e2b5aab9ccbd0840eb17a280699 73.32kB / 73.32kB                                                             924.5s

Edit: In my case I was accidentally using docker bundling because I forgot to install esbuild with npm. I'm not sure if my initial comment could still be relevant so i'll leave it up.

[whereisaaron]

CDK CLI Version: 2.74.0 Node.js Version: v18.15.0 OS: Windows 11 Docker: Rancher Desktop 1.8.1 CDE: Visual Studio Code: 1.77.3

I get the same issue too, bundling that normally takes 5 seconds suddenly takes 4-10 minutes just sitting there. For the same project I haven't seen this under Linux, only when using WSL2 under Windows 11.

My only work around right now is to stop Rancher Desktop, completely restart WSL2 by Restarting the 'LxssManager' Windows Service, the start Rancher Desktop again. After that docker bundling will work at normal speed for a few builds before eventually going funny again.

My (total guess) suspicion is this is related to how bundling is done by CDK. Instead of a normal docker build where the files are copied into the container, CDK directly mounts the build files folder into the container. I suspect some Windows/WSL2/docker file mounting drama eventually causes severe delays to file operations.

I am also suspicious of Visual Studio Code's WSL support. Sometime I can quickly fix the problem just by deleting the 'bin' and 'obj' folders from the build directory that CDL mounts into docker. But that is very hit or miss, can't tie it down yet.

[metuuu]

I'm having the same issue.

[sladg]

Facing same issue on latest versions on CDK 2.101.1 and containerised app. Nothing huge really, just simple Node app.