search

serverless-nextjs / serverless-next.js

⚡ Deploy your Next.js apps on AWS Lambda@Edge via Serverless Components
MIT License
4.45k stars 455 forks source link

RequestEntityTooLargeException: Request must be smaller than 69905067 bytes for the CreateFunction operation #236

Closed jschimmoeller closed 3 years ago

jschimmoeller commented 4 years ago

Hey daniel,

I finally had some time to check the latest code against my large next.js project and I am get this error;

error: RequestEntityTooLargeException: Request must be smaller than 69905067 bytes for the CreateFunction operation at Object.extractError (C:\Users\james.serverless\components\registry\npm\serverless-next.js@1.5.0\node_modules\serverless-next.js\node_modules\aws-sdk\lib\protocol\json.js:51:27) at Request.extractError (C:\Users\james.serverless\components\registry\npm\serverless-next.js@1.5.0\node_modules\serverless-next.js\node_modules\aws-sdk\lib\protocol\rest_json.js:55:8) at Request.callListeners (C:\Users\james.serverless\components\registry\npm\serverless-next.js@1.5.0\node_modules\serverless-next.js\node_modules\aws-sdk\lib\sequential_executor.js:106:20) at Request.emit (C:\Users\james.serverless\components\registry\npm\serverless-next.js@1.5.0\node_modules\serverless-next.js\node_modules\aws-sdk\lib\sequential_executor.js:78:10) at Request.emit (C:\Users\james.serverless\components\registry\npm\serverless-next.js@1.5.0\node_modules\serverless-next.js\node_modules\aws-sdk\lib\request.js:683:14) at Request.transition (C:\Users\james.serverless\components\registry\npm\serverless-next.js@1.5.0\node_modules\serverless-next.js\node_modules\aws-sdk\lib\request.js:22:10) at AcceptorStateMachine.runTo (C:\Users\james.serverless\components\registry\npm\serverless-next.js@1.5.0\node_modules\serverless-next.js\node_modules\aws-sdk\lib\state_machine.js:14:12) at C:\Users\james.serverless\components\registry\npm\serverless-next.js@1.5.0\node_modules\serverless-next.js\node_modules\aws-sdk\lib\state_machine.js:26:10 at Request. (C:\Users\james.serverless\components\registry\npm\serverless-next.js@1.5.0\node_modules\serverless-next.js\node_modules\aws-sdk\lib\request.js:38:9) at Request. (C:\Users\james.serverless\components\registry\npm\serverless-next.js@1.5.0\node_modules\serverless-next.js\node_modules\aws-sdk\lib\request.js:685:12) { message: 'Request must be smaller than 69905067 bytes for the CreateFunction operation', code: 'RequestEntityTooLargeException', time: 2019-11-12T14:54:17.708Z, requestId: '84af6699-36b9-4a2b-ab1c-88c468b47d19', statusCode: 413, retryable: false, retryDelay: 86.11399872761147 }

135s » nextApp » RequestEntityTooLargeException: Request must be smaller than 69905067 bytes for the CreateFunction operation

nodabladam commented 4 years ago

I am slightly terrified of my Next.js app growing to the point where it exceeds the max direct upload size. I bought this theme and rolled it out to serverless as my first test of serverless and got a similar error.

Perhaps we can have a flag that enables deployment using the s3 zip method which has a higher size? https://hackernoon.com/exploring-the-aws-lambda-deployment-limits-9a8384b0bec3

danielcondemarin commented 4 years ago

@jschimmoeller could you provide what is inside your .serverless_nextjs directory including sizes please?

jschimmoeller commented 4 years ago

@danielcondemarin ok as you will know there are 2 directories in this folder.

api-lambda - pages/api - i have 52 items with the average size 6.9MB image

default-lambda - pages - i have 23 items with the average size 8.2MB

image

also if i zip up these directories the sizes are api-lambda 50 mb default-lambda 41mb

image image

danielcondemarin commented 4 years ago

There is a lot of duplication across all page bundles when building using the next serverless target. There should be a way to output the common chunks across page bundles in a separate file which would greatly reduce the lambda artifact size. That would involve going deep into the webpack config that next implements for the serverless target though. Happy to listen to any other suggestions like @jschimmoeller ’s

nodabladam commented 4 years ago

Related side note...I noticed an issue open that next.js serverless doesn't enable treeshaking: https://github.com/zeit/next.js/issues/8956

Seems strange to me given all the focus and performance improvements lately Next seems to be putting into serverless but perhaps it is true? Would something like this have any positive effect? https://www.npmjs.com/package/serverless-plugin-optimize

hnprashanth commented 4 years ago

Would something like this have any positive effect? https://www.npmjs.com/package/serverless-plugin-optimize

Tried this plugin, didn't help

Enalmada commented 4 years ago

If you put this in your webpack config, it might help as a workaround until treeshaking is fixed in Next.js serverless (https://github.com/zeit/next.js/issues/8956)

      config.optimization.minimize = true;

(Note that this might increase build time so you would want to make this only happen during production build...probably using using phases.)

vishwasnavadak commented 4 years ago

If anybody is facing this issue has opted out of Automatic Static Optimisation in _app.js and if your app has few pages which can be statically rendered, then try removing the getInitialProps from _app.js and place it in the pages which needs the SSR. This will decrease the bundle size.

danielcondemarin commented 4 years ago

If anybody is facing this issue has opted out of Automatic Static Optimisation in _app.js and if your app has few pages which can be statically rendered, then try removing the getInitialProps from _app.js and place it in the pages which needs the SSR. This will decrease the bundle size.

Great advice! With the next component you can have an indefinite number of static pages as they’re stored in S3. On the other hand, SSR pages are zipped up and used in lambda@edge so the lambda artifact size limit applies there.

jschimmoeller commented 4 years ago

thanks for the recommendation; i did try both and each still failed.

  1. config.optimization.minimize = true; was able to build it via next (which took a very long time 15 plus minutes) however when I ran it via "serverless" i get the following error:

`

C:\Users\james\projects\new-refiner>serverless

error: Error: Command failed with exit code 1 (Unknown system error -1): node_modules/.bin/next build at makeError (C:\Users\james.serverless\components\registry\npm\serverless-next.js@1.6.1\node_modules\serverless-next.js\node_modules\execa\lib\error.js:59:11) at handlePromise (C:\Users\james.serverless\components\registry\npm\serverless-next.js@1.6.1\node_modules\serverless-next.js\node_modules\execa\index.js:112:26) at processTicksAndRejections (internal/process/task_queues.js:85:5) { command: 'node_modules/.bin/next build', exitCode: 1, exitCodeName: 'Unknown system error -1', stdout: 'Creating an optimized production build...', stderr: 'events.js:180\r\n' + " throw er; // Unhandled 'error' event\r\n" + ' ^\r\n' + '\r\n' + 'Error: write UNKNOWN\r\n' + ' at ChildProcess.target._send (internal/child_process.js:806:20)\r\n' + ' at ChildProcess.target.send (internal/child_process.js:676:19)\r\n' + ' at ChildProcessWorker.send (C:\Users\james\projects\new-refiner\node_modules\jest-worker\build\workers\ChildProcessWorker.js:286:17)\r\n' + ' at WorkerPool.send (C:\Users\james\projects\new-refiner\node_modules\jest-worker\build\WorkerPool.js:32:34)\r\n' + ' at Farm._process (C:\Users\james\projects\new-refiner\node_modules\jest-worker\build\Farm.js:129:10)\r\n' + ' at onEnd (C:\Users\james\projects\new-refiner\node_modules\jest-worker\build\Farm.js:122:12)\r\n' + ' at ChildProcessWorker._onProcessEnd (C:\Users\james\projects\new-refiner\node_modules\jest-worker\build\workers\ChildProcessWorker.js:280:14)\r\n' + ' at ChildProcessWorker.onMessage (C:\Users\james\projects\new-refiner\node_modules\jest-worker\build\workers\ChildProcessWorker.js:220:14)\r\n' + ' at ChildProcess.emit (events.js:203:13)\r\n' + ' at emit (internal/child_process.js:876:12)\r\n' + "Emitted 'error' event at:\r\n" + ' at internal/child_process.js:810:39\r\n' + ' at processTicksAndRejections (internal/process/task_queues.js:75:11) {\r\n' + " errno: 'UNKNOWN',\r\n" + " code: 'UNKNOWN',\r\n" + " syscall: 'write'\r\n" + '}', all: 'Creating an optimized production build...\n' + 'events.js:180\r\n' + " throw er; // Unhandled 'error' event\r\n" + ' ^\r\n' + '\r\n' + 'Error: write UNKNOWN\r\n' + ' at ChildProcess.target._send (internal/child_process.js:806:20)\r\n' + ' at ChildProcess.target.send (internal/child_process.js:676:19)\r\n' + ' at ChildProcessWorker.send (C:\Users\james\projects\new-refiner\node_modules\jest-worker\build\workers\ChildProcessWorker.js:286:17)\r\n' + ' at WorkerPool.send (C:\Users\james\projects\new-refiner\node_modules\jest-worker\build\WorkerPool.js:32:34)\r\n' + ' at Farm._process (C:\Users\james\projects\new-refiner\node_modules\jest-worker\build\Farm.js:129:10)\r\n' + ' at onEnd (C:\Users\james\projects\new-refiner\node_modules\jest-worker\build\Farm.js:122:12)\r\n' + ' at ChildProcessWorker._onProcessEnd (C:\Users\james\projects\new-refiner\node_modules\jest-worker\build\workers\ChildProcessWorker.js:280:14)\r\n' + ' at ChildProcessWorker.onMessage (C:\Users\james\projects\new-refiner\node_modules\jest-worker\build\workers\ChildProcessWorker.js:220:14)\r\n' + ' at ChildProcess.emit (events.js:203:13)\r\n' + ' at emit (internal/child_process.js:876:12)\r\n' + "Emitted 'error' event at:\r\n" + ' at internal/child_process.js:810:39\r\n' + ' at processTicksAndRejections (internal/process/task_queues.js:75:11) {\r\n' + " errno: 'UNKNOWN',\r\n" + " code: 'UNKNOWN',\r\n" + " syscall: 'write'\r\n" + '}', failed: true, timedOut: false, isCanceled: false, killed: false, signal: undefined }

491s » nextApp » Error: Command failed with exit code 1 (Unknown system error -1): node_modules/.bin/next build `

  1. since the optimization was causing an issue, I commented it out and removed from _app.js the getInitialProps method; it failed with the following error:

` C:\Users\james\projects\new-refiner>serverless

error: RequestEntityTooLargeException: Request must be smaller than 69905067 bytes for the CreateFunction operation at Object.extractError (C:\Users\james.serverless\components\registry\npm\serverless-next.js@1.6.1\node_modules\serverless-next.js\node_modules\aws-sdk\lib\protocol\json.js:51:27) at Request.extractError (C:\Users\james.serverless\components\registry\npm\serverless-next.js@1.6.1\node_modules\serverless-next.js\node_modules\aws-sdk\lib\protocol\rest_json.js:55:8) at Request.callListeners (C:\Users\james.serverless\components\registry\npm\serverless-next.js@1.6.1\node_modules\serverless-next.js\node_modules\aws-sdk\lib\sequential_executor.js:106:20) at Request.emit (C:\Users\james.serverless\components\registry\npm\serverless-next.js@1.6.1\node_modules\serverless-next.js\node_modules\aws-sdk\lib\sequential_executor.js:78:10) at Request.emit (C:\Users\james.serverless\components\registry\npm\serverless-next.js@1.6.1\node_modules\serverless-next.js\node_modules\aws-sdk\lib\request.js:683:14) at Request.transition (C:\Users\james.serverless\components\registry\npm\serverless-next.js@1.6.1\node_modules\serverless-next.js\node_modules\aws-sdk\lib\request.js:22:10) at AcceptorStateMachine.runTo (C:\Users\james.serverless\components\registry\npm\serverless-next.js@1.6.1\node_modules\serverless-next.js\node_modules\aws-sdk\lib\state_machine.js:14:12) at C:\Users\james.serverless\components\registry\npm\serverless-next.js@1.6.1\node_modules\serverless-next.js\node_modules\aws-sdk\lib\state_machine.js:26:10 at Request. (C:\Users\james.serverless\components\registry\npm\serverless-next.js@1.6.1\node_modules\serverless-next.js\node_modules\aws-sdk\lib\request.js:38:9) at Request. (C:\Users\james.serverless\components\registry\npm\serverless-next.js@1.6.1\node_modules\serverless-next.js\node_modules\aws-sdk\lib\request.js:685:12) { message: 'Request must be smaller than 69905067 bytes for the CreateFunction operation', code: 'RequestEntityTooLargeException', time: 2019-12-02T14:41:00.854Z, requestId: '9b4fb4c9-df53-4ab8-acd9-9b15fefe597e', statusCode: 413, retryable: false, retryDelay: 13.134552790061571 }

94s » nextApp » RequestEntityTooLargeException: Request must be smaller than 69905067 bytes for the CreateFunction operation

C:\Users\james\projects\new-refiner>
`

kartikag01 commented 4 years ago

I have tried modifying webpack config to, 

config.optimization.splitChunks.cacheGroups = { }

and this worked,

But what i want config is 

    config.optimization.splitChunks.cacheGroups = {
      vendor: {
        name: "vendor",
        test: /[\\/]node_modules[\\/](react|react-dom|my-package)[\\/]/,
        chunks: "all",
        enforce: true,
      },
    };

this starts to gives error RequestEntityTooLargeException.

danielcondemarin commented 4 years ago

FYI - Next.js recently added a new build target called "serverless trace". I haven't had time to look into it properly but looks like it would solve this issue. https://github.com/zeit/next.js/pull/8246

If someone has time to do some initial investigations that would be much appreciated 🙏

Enalmada commented 4 years ago

I do think that serverless trace could make all serverless-next pages load faster because everything would be so much smaller https://github.com/zeit/next.js/issues/8956#issuecomment-570434649

danielcondemarin commented 4 years ago

I do think that serverless trace could make all serverless-next pages load faster because everything would be so much smaller zeit/next.js#8956 (comment)

I am very keen on getting this working. It would not just solve this particular issue with the size of the deployment artefact but also speed up deployments considerably. In terms of page load I don't think there will be much (if any) improvements as this only affects server side artefact size.

@Enalmada Have you tried using the serverless trace with this project? It may give us a rough idea of what needs to be done to support it.

danielcondemarin commented 4 years ago

So I've been doing a bit of investigation around this already.

The serverless-trace target does not transpile node_modules via webpack like theserverless target does. This is what leads to smaller page bundles which is great.

What this means is this project has to zip up the node_modules and upload that to Lambda@Edge. The good news is that's exactly what @now/next already does here so we could replicate it. Also, we can use https://github.com/zeit/node-file-trace to track exactly which node_modules need zipping up!

Enalmada commented 4 years ago

I tried passing serverless-trace but quickly realized the errors were just a bit beyond my ability. In terms of performance I was thinking of cold starts...that server side artifact being smaller might help with that.

danielcondemarin commented 4 years ago

I have got a working proof of concept using the approach I mentioned in my comment above. This is what Zeit NOW uses in their stack as well so I'm considering making experimental-serverless-trace the default target. Watch this space I should have something ready for testing soon.

mertyildiran commented 4 years ago

I also got this error when I tried to deploy ~4000 pages website into AWS Lambda:

  error:
  { RequestEntityTooLargeException: Request must be smaller than 69905067 bytes for the UpdateFunctionCode operation
    at Object.extractError (/home/mertyildiran/.serverless/components/registry/npm/serverless-next.js@1.11.3/node_modules/aws-sdk/lib/protocol/json.js:51:27)
    at Request.extractError (/home/mertyildiran/.serverless/components/registry/npm/serverless-next.js@1.11.3/node_modules/aws-sdk/lib/protocol/rest_json.js:55:8)
    at Request.callListeners (/home/mertyildiran/.serverless/components/registry/npm/serverless-next.js@1.11.3/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/home/mertyildiran/.serverless/components/registry/npm/serverless-next.js@1.11.3/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/home/mertyildiran/.serverless/components/registry/npm/serverless-next.js@1.11.3/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/home/mertyildiran/.serverless/components/registry/npm/serverless-next.js@1.11.3/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/home/mertyildiran/.serverless/components/registry/npm/serverless-next.js@1.11.3/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /home/mertyildiran/.serverless/components/registry/npm/serverless-next.js@1.11.3/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/home/mertyildiran/.serverless/components/registry/npm/serverless-next.js@1.11.3/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/home/mertyildiran/.serverless/components/registry/npm/serverless-next.js@1.11.3/node_modules/aws-sdk/lib/request.js:685:12)
  message:
   'Request must be smaller than 69905067 bytes for the UpdateFunctionCode operation',
  code: 'RequestEntityTooLargeException',
  time: 2020-05-07T05:24:59.307Z,
  requestId: 'dc34cd8b-1215-4f50-a55f-61ce277c4af1',
  statusCode: 413,
  retryable: false,
  retryDelay: 74.52977207528329 }

There are actually 30 - 40 serverless functions in our application. By page, I mean the different pages created by dynamic routes which only contiant getStaticProps and getStaticPaths. When I tried ~500 pages it didn't exceed this limit. But when I tried ~4000 pages it exceeded the limit. Clearly the function size is increasing by the dynamic page count which should not happen.

danielcondemarin commented 4 years ago

There are actually 30 - 40 serverless functions in our application. By page, I mean the different pages created by dynamic routes which only contiant getStaticProps and getStaticPaths. When I tried ~500 pages it didn't exceed this limit. But when I tried ~4000 pages it exceeded the limit. Clearly the function size is increasing by the dynamic page count which should not happen.

The function size will increase depending on how many server side rendered pages you have. Unfortunately next's serverless target doesn't dedup common dependencies across page bundles, which is why they've introduced a new target, the serverless-trace which I will be adding pretty soon to the project. It still outputs page bundles but common dependencies in the node_modules are marked as externals in webpack, so there is no duplication.

mertyildiran commented 4 years ago

@danielcondemarin OK, let me rephrase what I said:

With serverless function, I mean: pages/post/[id].js

and with page, I mean:

/post/1.html
/post/2.html
...
/post/3999.html
/post/4000.html

If there are 500 posts in the database, I can deploy the website successfully. But if there are 4000 posts in the database then I get RequestEntityTooLargeException and deployment fails.

So I don't see any relation between node_modules or webpack and the number of posts in my database. I mean how does it create duplicate dependencies for /post/1.html and /post/2.html?

danielcondemarin commented 4 years ago

@mertyildiran I understand now. They're all prerendered static pages. They should not end up in the lambda@edge artefact. Could you check your .serverless_nextjs/default-lambda folder. It shouldn't have html pages in it. Same goes for .serverless_nextjs/api-lambda

mertyildiran commented 4 years ago

@danielcondemarin there are no HTML files but in .serverless_nextjs/default-lambda there are tons of prerendered JSON files. I think they don't suppose to be there?

mertyildiran commented 4 years ago

@danielcondemarin I think we should add a control like !isJSONPage to here. Let me do that, I'm preparing another PR.

danielcondemarin commented 4 years ago

@danielcondemarin there are no HTML files but in .serverless_nextjs/default-lambda there are tons of prerendered JSON files. I think they don't suppose to be there?

Good catch! You're right they shouldn't be there.

This only became an issue after getStaticProps was introduced.

danielcondemarin commented 4 years ago

Hi folks 👋

Some good news, I've managed to get the new serverless-trace target working on serverless-next.js! It means node_modules are not bundled anymore within the page bundles generated by next build. Instead node_modules are treated as externals (in webpack) therefore the page bundles don't contain duplicated dependencies code. There are still some optimisations I could make, like compressing the node_modules, same to what Now does. However I am hoping with the current implementation it's good enough and solve this issue altogether.

Ideally this will be tested with a large app. could I have some volunteers please? It is currently released in serverless-next.js@1.12.0-alpha.4. All you'd need to do is make sure you are targeting experimental-serverless-trace in your next.config.js:

module.exports = {
  target: "experimental-serverless-trace",
  ...
}
# serverless.yml
yourApp:
  component:  serverless-next.js@1.12.0-alpha.4
  ...

Don't be put off by the target being "experimental". This is actually what Now uses under the hood for serverless deployments. I've pretty much replicated the same the now-next builder does.

Reference PR: https://github.com/danielcondemarin/serverless-next.js/pull/405

thiagozf commented 4 years ago

@danielcondemarin Thank you for this awesome feature! I've tested serverless-next.js@1.12.0-alpha.4 with a large(ish) application and the default lambda size went from 23.1 MB down to 4.0 MB. However, I had to do some minor tweaks to get it working.

PS.: I'm not sure if I should open separate issues, so I'll just report the said tweaks below.

1. next.config.js is not normalized There is a new function called isServerlessTraceTarget that checks if we are using experimental-serverless-trace. However, one can also export a function from next.config.js (see docs) and we are not considering this scenario. To get it working, I've patched the function to return true, but ideally we should normalize the config import, like Next.js does here.

2. Next config dir is being used as the base path for "@zeit/node-file-trace" The Next.js application I'm working on is part of a monorepo. Its dependencies (node_modules) are not located on ${nextConfigDir}, but on the repository root. Since the base param is being set to ${nextConfigDir} here and here, @zeit/node-file-trace was ignoring all my dependencies :(. ~Since I'm triggering my deploys from the repo root, I've removed the base param and used the default process.cwd(), but I don't know if this is the best approach to fix this case.~ EDIT: I think this is a problem with @zeit/node-file-trace and should be fixed there. I'm going to send a PR.

3. Wrong API Lambda referenced on CloudFront distribution I don't think this one is related to #405, but probably to #282. When I tried to deploy my app for the first time, the component threw an error saying that "Setting custom cache behaviour for api/ route "api" is not supported". I had to change my CloudFront config on serverless.yml from api to api/* and then the app was deployed successfully. However, my CloudFront distribution was now pointing the api/* path to the default Lambda instead of the API Lambda 🤔. I think it is happening because the default Lambda CloudFront config overwrites it here. I'm not sure how I'm supposed to configure the API's CloudFront distribution now, but I need to forward some headers (Authorization, Accept, etc) to the API handlers. Any tips here?

danielcondemarin commented 4 years ago

@thiagozf Great write up! Thanks a lot for feeding back your findings!

  1. next.config.js is not normalized There is a new function called isServerlessTraceTarget that checks if we are using experimental-serverless-trace. However, one can also export a function from next.config.js (see docs) and we are not considering this scenario. To get it working, I've patched the function to return true, but ideally we should normalize the config import, like Next.js does here.

Good catch! That should be an easy fix, although I'm more inclined to go for a solution where serverless-next.js deals with the target so users don't even need to specify it. The now-next builder implements that here so we could do the same.

  1. Next config dir is being used as the base path for "@zeit/node-file-trace"

I could change it to process.cwd() I think that should cover all cases (i.e. being in a monorepo or not).

  1. Wrong API Lambda referenced on CloudFront distribution

Yeah that sounds like an issue introduced in https://github.com/danielcondemarin/serverless-next.js/pull/282. @stan-sack may be able to help with this?

thiagozf commented 4 years ago

  1. Agree, it is better if we don't need to specify it.

  2. Initially, I thought that changing it to process.cwd() would be enough to fix it too, but there is a "problem": chunks are going to be copied to .serverless_nextjs/<lambda>/[packages/<package>/].next/serverless (instead of .serverless_nextjs/<lambda>/.next/serverless) and the Next build references them with something like ../serverless/<chunk>, because it uses the config dir as the base path 🤦. I have not found a good solution for this.

image (app-admin is my Next app package)

danielcondemarin commented 4 years ago
  1. Initially, I thought that changing it to process.cwd() would be enough to fix it too, but there is a "problem": chunks are going to be copied to .serverless_nextjs/<lambda>/[packages/<package>/].next/serverless (instead of .serverless_nextjs/<lambda>/.next/serverless) and the Next build references them with something like ../serverless/<chunk>, because it uses the config dir as the base path 🤦. I have not found a good solution for this.

image (app-admin is my Next app package)

path.relative should help here. Something along the lines of:

// dotNextDirectory: packages/app-admin/.next
// filePath: packages/app-admin/.next/serverless/xyz.js

const dst = path.relative(dotNextDirectory, filePath);

// dst: serverless/xyz.js

fse.copy(
   filePath,
   join(this.outputDir, DEFAULT_LAMBDA_CODE_DIR, dst)
);
Belco90 commented 4 years ago

I ran into this issue this morning. I updated the project as Daniel mentioned with latest alpha version but something else is failing now.

The deployment seems to work fine, but when I check the actual webapp I get a 503 error. This is what I'm getting in CloudWatch:

{
    "errorType": "Error",
    "errorMessage": "Cannot find module 'next/dist/next-server/server/render'",
    "code": "MODULE_NOT_FOUND",
    "stack": [
        "Error: Cannot find module 'next/dist/next-server/server/render'",
        "    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)",
        "    at Function.Module._load (internal/modules/cjs/loader.js:562:25)",
        "    at Module.require (internal/modules/cjs/loader.js:692:17)",
        "    at require (internal/modules/cjs/helpers.js:25:18)",
        "    at Object.KK5V (/var/task/pages/index.js:1178:18)",
        "    at __webpack_require__ (/var/task/pages/index.js:23:31)",
        "    at Module.MzBc (/var/task/pages/index.js:1406:28)",
        "    at __webpack_require__ (/var/task/pages/index.js:23:31)",
        "    at /var/task/pages/index.js:91:18",
        "    at Object.<anonymous> (/var/task/pages/index.js:94:10)"
    ]
}

I'm little bit lost after this last error. I'm using:

I had to opt-out Automatic Static Optimization because of the redux wrapper the app needs, so I can't try to do a workaround for removing the getInitialProps from my _app for now. I'll check other possible workarounds I saw here meanwhile.

Thanks.

Belco90 commented 4 years ago

Additionally, I can see this warning in the console only when installing last serverless-next.js alpha:

@sls-next/lambda-at-edge@1.2.0-alpha.1 requires a peer of next-aws-cloudfront@file:../cloudfront-lambda@edge-compat but none is installed. You must install peer dependencies yourself.

danielcondemarin commented 4 years ago

@thiagozf I've just pushed a fix that should solve the issue you were seeing in your monorepo setup. Could you test using @serverless-next.js@1.12.0-alpha.5 please? I've also published a fix for the CloudFront config. issues you were seeing. credits to @stan-sack for this!

@Belco90 I suspect the new release may also fix the issue you encountered (Cannot find module 'next/..). I've also pushed a fix for the misleading peer dep warning 👍

Belco90 commented 4 years ago

@danielcondemarin Thanks, amazing work! I'll try to check it at some point this week.

kenberkeley commented 4 years ago

I tried serverless-next.js@1.12.0-alpha.6 and got Cannot find module XXX as @Belco90 did

It seems that we need to bundle node_modules to S3?

@danielcondemarin Any quick workaround for this issue?

kenberkeley commented 4 years ago

I added my "copy node_modules" logic next to

https://github.com/danielcondemarin/serverless-next.js/blob/57142970b08e6bc3faf0fc70749b3b0501ad7869/packages/lambda-at-edge/src/build.ts#L191-L198

And ... it took forever to deploy... Because... all the files were uploaded one by one instead of zip 😂😂😂

I realised this because AWS sent me an email:

image

kenberkeley commented 4 years ago

Update: the Lambda@Edge will be deployed to N. Virginia (us-east-1) so that we can upload zip there.

danielcondemarin commented 4 years ago

I tried serverless-next.js@1.12.0-alpha.6 and got Cannot find module XXX as @Belco90 did

It seems that we need to bundle node_modules to S3?

@danielcondemarin Any quick workaround for this issue?

@kenberkeley That's exactly what serverless-next.js does. It finds what's actually required by the pages to server side render and copies those dependencies into the lambda artefact. You don't need to manually zip & upload node_modules.

Could you expand which node_module is missing and how is being used?

kenberkeley commented 4 years ago

@danielcondemarin All, including the very basic react... There is only next-aws-cloudfront in the node_modules...

danielcondemarin commented 4 years ago

What's your serverless.yml like? Make sure you specify the alpha version for this to work:

# serverless.ml
component: serverless-next.js@1.12.0-alpha.6
...
kenberkeley commented 4 years ago

Yes, I did that.

my-app:
  component: serverless-next.js@1.12.0-alpha.6
  inputs:
    domain: ['www', 'domain.com']
danielcondemarin commented 4 years ago

And just one more check, did you specify experimental-serverless-trace in your next.config.js ? 

// next.config.js
module.exports = {
  target: "experimental-serverless-trace"
}

I'm hoping to make this automatically handled by the component but for now you need to specify it.

kenberkeley commented 4 years ago

@danielcondemarin Yes, I did that as well...

danielcondemarin commented 4 years ago

@danielcondemarin Yes, I did that as well...

I'll take a look later but with that configuration I would expect it to copy the node_modules. Also, if you could provide a simple repro that would be great 🙏

kenberkeley commented 4 years ago

By the way, I am using Yarn workspace (monorepo)...

cuckoointernet commented 4 years ago

@danielcondemarin Thanks for your contribution mate! I've ran into the request issue in my project when I added 1 too many pages.

Your method seems to work, I get a clean upload however I seem to be getting weird rendering issues, it looks like the SSR version goes up clean without any issue (disabling javascript shows the page rendering correctly) then as soon as the injected javascript is executed on the client everything turns to shit (pardon the language).

Could this be ordering of modules that are being loaded, or is it something similar to what other people are experiencing here with the node_modules not being uploaded?

danielcondemarin commented 4 years ago

I'd appreciate if

@danielcondemarin Thanks for your contribution mate! I've ran into the request issue in my project when I added 1 too many pages.

Your method seems to work, I get a clean upload however I seem to be getting weird rendering issues, it looks like the SSR version goes up clean without any issue (disabling javascript shows the page rendering correctly) then as soon as the injected javascript is executed on the client everything turns to shit (pardon the language).

Could this be ordering of modules that are being loaded, or is it something similar to what other people are experiencing here with the node_modules not being uploaded?

I need more information to give you an answer. If you provide a simple reproduction of the problem in a repo I will be able to easily replicate and fix it. Some of these issues require a lot of context to be debugged, e.g. deploying in a monorepo or not, next.config being used etc. The more info I get when you find an issue the sooner I can fix it and we can get this feature out of alpha and put onto stable 🙏 PS. That's meant for everyone reporting issues and not just @cuckoointernet btw 🙂

cuckoointernet commented 4 years ago

Yeah totally, I'll get something up for you this afternoon.

krish-dev commented 4 years ago

Thanks, @danielcondemarin that you consider for experimental-serverless-trace support.

I'm getting a couple of issues

  1. 503 ERROR for my index route which is an SSR enabled page. However, it works perfectly for optimized route which is an SSG page.
  2. Lambda not generated for dynamic Catch all routes (products/[...magento_route].jsx)

Below I'm sharing details of the configuration and generated files. I'm not sure how to deal with node_modules, checked s3 bucket. node_modules are not there.

Here is my yml file

myapp:
  component: serverless-next.js@1.12.0-alpha.6
  inputs:
    name: myapp-pwa-prod
    memory: 512
    bucketName: myapp-pwa-prod
target: 'experimental-serverless-trace'

This is my applications routes (pages)

├── _app.jsx
├── _document.jsx
├── index.jsx ======> SSR 
├── optimized.jsx ======> SSG
└── products
    ├── [...magento_route].jsx ======> Dynamic:Empty
    └── index.jsx ======> NonDynamic:Empty

This is the generated files by serverless-next.js@1.12.0-alpha.6

└── default-lambda
    ├── index.js
    ├── manifest.json
    ├── node_modules
    │   └── next-aws-cloudfront
    │       └── index.js
    ├── pages
    │   ├── _error.js
    │   ├── index.js
    │   ├── optimized.js
    │   └── products
    └── prerender-manifest.json

Genarated manifest.json

{
    "pages": {
        "ssr": {
            "dynamic": {},
            "nonDynamic": {
                "/_error": "pages/_error.js",
                "/index": "pages/index.js",
                "/optimized": "pages/optimized.js",
                "/": "pages/index.js",
                "/_app": "pages/_app.js",
                "/_document": "pages/_document.js"
            }
        },
        "html": {
            "dynamic": {
                "/products/:magento_route*": {
                    "file": "pages/products/[...magento_route].html",
                    "regex": "^\\/products(?:\\/((?:[^\\/#\\?]+?)(?:\\/(?:[^\\/#\\?]+?))*))?[\\/#\\?]?$"
                }
            },
            "nonDynamic": {
                "/products": "pages/products.html",
                "/404": "pages/404.html"
            }
        }
    },
    "publicFiles": {...}
}

Genarated prerender-manifest.json

{
    "version": 2,
    "routes": {
        "/optimized": {
            "initialRevalidateSeconds": false,
            "srcRoute": null,
            "dataRoute": "/_next/data/q_9DzZBBBUtp7GgQICQaU/optimized.json"
        }
    },
    "dynamicRoutes": {},
    "preview": {...}
}
cuckoointernet commented 4 years ago

@danielcondemarin Thanks for your contribution mate! I've ran into the request issue in my project when I added 1 too many pages.

Your method seems to work, I get a clean upload however I seem to be getting weird rendering issues, it looks like the SSR version goes up clean without any issue (disabling javascript shows the page rendering correctly) then as soon as the injected javascript is executed on the client everything turns to shit (pardon the language).

Could this be ordering of modules that are being loaded, or is it something similar to what other people are experiencing here with the node_modules not being uploaded?

Sorry I didn't come through with an example, I figured out the issue when I was creating the sample. Basically, if you use a headless cms such as contentful or prismic, make sure you load as staticProps over initialProps, the changeover of this within next happened during development so wasn't picked up till now. Still have some investigation to get to the bottom of why this was happening exactly, but more than likely due to unintended side-effects from 9.2->9.4 so absolutely nothing to do with what you have done here @danielcondemarin . Again, thanks for this change, our deployments now take 63seconds and the bloat that existed there previously doesn't exist anymore :) 👍

krish-dev commented 4 years ago

@danielcondemarin adding more observations to my previous comment.

Also, I have a few Dynamic imports with no SSR, which generates some chunks in case of the target as experimental-serverless-trace. I found these file inside ./next/serverless directory. which is also not uploaded to AWS. shared the tree below for ref.

.
├── 7.7b867ad9435063b171d7.js =====> chunk file
├── 8.27fb944afb2b87d03e83.js =====> chunk file
├── 9.b7b016a1635f6fea509f.js =====> chunk file
├── init-server.js
├── on-error-server.js
├── pages
│   ├── 404.html
│   ├── _error.js
│   ├── index.js
│   ├── optimized.html
│   ├── optimized.js
│   ├── optimized.json
│   ├── products
│   │   └── [...magento_route].html
│   └── products.html
└── pages-manifest.json