fix: upload to s3 in chunks

[jadenv]

When uploading a site with hundreds of static assets, the custom lambda resource that uploads to S3 will error out with Error: EMFILE: too many open files.

This is due to a read stream being opened for every static upload concurrently.

This PR batches the uploads into more manageable chunk sizes so there isn't an limitless number of file pointers opened at once.

[jadenv]

@bestickley I'm not exactly sure how to create a release in my fork that I can test in my project having this issue, any pointers?

[bestickley]

thank you, @jadenv! You should be able to run yarn compile and then link your local CDK project (with CDK app you want to deploy) to this repo to test out. Does that work for you?

[bestickley]

also, have you tried increasing memory? with overrides? would that help?

[jadenv]

@bestickley Yep, increasing memory size was the first thing I tried! No dice however. We have close to 1000 images as part of our site, so I would imagine at some point even with more memory this would still be an issue unless there is some kind of chunk limit in place.

I'll give that yarn compile a try 👍🏻

[jadenv]

@bestickley I got stuck on this error when trying to link our project.

Type 'import("~/next/node_modules/constructs/lib/construct").Node' is not assignable to type 'import("~/cdk-nextjs/node_modules/constructs/lib/construct").Node'.
Types have separate declarations of a private property 'host'.

32     const nextJs = new Nextjs(this, 'NextLambdaEdge', {

Maybe when you have time you could try it with one of your projects and then once there is an npm release created I will have more luck when trying that.

[bestickley]

@jadenv, that looks like a type issue between different versions of CDK. Can you ignore it and still deploy?

[jadenv]

@bestickley Ah yep, I was able to ignore that type issue, however ran into what appears to be an open-next dependency issue now. I double checked those AWS dependencies are listed in the open-next package.json, so not sure if this is maybe because of trying to yarn link with open-next's pnpm packages or something?

┌──────────────────────────────┐
│ OpenNext — Generating bundle │
└──────────────────────────────┘

Bundling static assets...
Bundling cache assets...
Bundling server function...
✘ [ERROR] Could not resolve "@aws-sdk/client-dynamodb"

    ../../open-next/packages/open-next/dist/adapters/server-adapter.js:1:31:
      1 │ import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
        ╵                                ~~~~~~~~~~~~~~~~~~~~~~~~~~

  The Yarn Plug'n'Play manifest forbids importing "@aws-sdk/client-dynamodb" here because it's not
  listed as a dependency of this package:

    ../../../.pnp.cjs:36:31:
      36 │         "packageDependencies": [\
         ╵                                ~~

  You can mark the path "@aws-sdk/client-dynamodb" as external to exclude it from the bundle, which
  will remove this error.

✘ [ERROR] Could not resolve "@aws-sdk/client-s3"

    ../../open-next/packages/open-next/dist/adapters/server-adapter.js:2:25:
      2 │ import { S3Client } from "@aws-sdk/client-s3";
        ╵                          ~~~~~~~~~~~~~~~~~~~~

  The Yarn Plug'n'Play manifest forbids importing "@aws-sdk/client-s3" here because it's not listed
  as a dependency of this package:

    ../../../.pnp.cjs:36:31:
      36 │         "packageDependencies": [\
         ╵                                ~~

  You can mark the path "@aws-sdk/client-s3" as external to exclude it from the bundle, which will
  remove this error.

✘ [ERROR] Could not resolve "@aws-sdk/client-sqs"

    ../../open-next/packages/open-next/dist/adapters/plugins/routing/util.js:1:46:
      1 │ import { SendMessageCommand, SQSClient } from "@aws-sdk/client-sqs";
        ╵                                               ~~~~~~~~~~~~~~~~~~~~~

  The Yarn Plug'n'Play manifest forbids importing "@aws-sdk/client-sqs" here because it's not listed
  as a dependency of this package:

    ../../../.pnp.cjs:36:31:
      36 │         "packageDependencies": [\
         ╵                                ~~

  You can mark the path "@aws-sdk/client-sqs" as external to exclude it from the bundle, which will
  remove this error.

✘ [ERROR] Could not resolve "path-to-regexp"

    ../../open-next/packages/open-next/dist/adapters/routing/matcher.js:1:31:
      1 │ import { compile, match } from "path-to-regexp";
        ╵                                ~~~~~~~~~~~~~~~~

  The Yarn Plug'n'Play manifest forbids importing "path-to-regexp" here because it's not listed as a
  dependency of this package:

    ../../../.pnp.cjs:36:31:
      36 │         "packageDependencies": [\
         ╵                                ~~

  You can mark the path "path-to-regexp" as external to exclude it from the bundle, which will
  remove this error.

~/open-next/packages/open-next/node_modules/esbuild/lib/main.js:1649
  let error = new Error(text);
              ^

Error: Build failed with 4 errors:
../../open-next/packages/open-next/dist/adapters/plugins/routing/util.js:1:46: ERROR: Could not resolve "@aws-sdk/client-sqs"
../../open-next/packages/open-next/dist/adapters/routing/matcher.js:1:31: ERROR: Could not resolve "path-to-regexp"
../../open-next/packages/open-next/dist/adapters/server-adapter.js:1:31: ERROR: Could not resolve "@aws-sdk/client-dynamodb"
../../open-next/packages/open-next/dist/adapters/server-adapter.js:2:25: ERROR: Could not resolve "@aws-sdk/client-s3"
    at failureErrorWithLog (~/open-next/packages/open-next/node_modules/esbuild/lib/main.js:1649:15)
    at ~/open-next/packages/open-next/node_modules/esbuild/lib/main.js:1058:25
    at runOnEndCallbacks (~/open-next/packages/open-next/node_modules/esbuild/lib/main.js:1484:45)
    at buildResponseToResult (~/open-next/packages/open-next/node_modules/esbuild/lib/main.js:1056:7)
    at ~/open-next/packages/open-next/node_modules/esbuild/lib/main.js:1085:16
    at responseCallbacks.<computed> (~/open-next/packages/open-next/node_modules/esbuild/lib/main.js:703:9)
    at handleIncomingPacket (~/open-next/packages/open-next/node_modules/esbuild/lib/main.js:762:9)
    at Socket.readFromStdout (~/open-next/packages/open-next/node_modules/esbuild/lib/main.js:679:7)
    at Socket.emit (node:events:517:28)
    at addChunk (node:internal/streams/readable:368:12) {
  errors: [Getter/Setter],
  warnings: [Getter/Setter]
}

[bestickley]

@jadenv, that's strange. You shouldn't need to install dependencies within the open-next git submodule. Quickest fix may be to just install dependencies within open-next folder with pnpm i. If that doesn't fix the issue, then you need to "unlink" the local open-next code from the open-next version that is being invoked within cdk-nextjs-standalone. The default build command is npx open-next@^2 build so maybe try using a different variation such that it doesn't reference local code?

[jadenv]

@bestickley i fiddled more with the pnpm packages and got it workin.

The thousand or so S3 assets for our site all uploaded and had our first successful deployment on v4 🎉

Should be set to merge as far as I can tell unless you have any comments!