aws-amplify / amplify-hosting

AWS Amplify Hosting provides a Git-based workflow for deploying and hosting fullstack serverless web applications.
https://aws.amazon.com/amplify/hosting/
Apache License 2.0
460 stars 116 forks source link

Standalone directory not found - NextJS 13 Amplify #3132

Closed ShiftySlothe closed 1 year ago

ShiftySlothe commented 1 year ago

Before opening, please confirm:

App Id

d3lq4oet4t4ooj

AWS Region

eu-west-1

Amplify Hosting feature

Frontend builds

Describe the bug

Standalone directory not found, please enable output standalone on your next.config.js file or set NEXT_PRIVATE_STANDALONE=true.

Output is standalone in next.config.js AND set to true as an ENV variable.

It appears that the build is finishing early for some reason.

On a single build, it could find the standalone output, but deemed the output to be too large.

Expected behavior

Build runs sucessfully.

Reproduction steps

Run build, await standalone directory not to be found.

Build Settings

version: 1
frontend:
  phases:
    preBuild:
      commands:
        - yarn install
    build:
      commands:
        - yarn run build
  artifacts:
    baseDirectory: .next
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*

Log output

``` # Put your logs below this line 2022-11-21T21:42:11.594Z [INFO]: # Cloning repository: git@github.com: 2022-11-21T21:42:15.799Z [INFO]: 2022-11-21T21:42:15.881Z [INFO]: Cloning into ''... 2022-11-21T21:42:15.882Z [INFO]: # Switching to commit: 2022-11-21T21:42:16.012Z [INFO]: Note: switching to ''. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by switching back to a branch. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -c with the switch command. Example: git switch -c Or undo this operation with: git switch - Turn off this advice by setting config variable advice.detachedHead to false HEAD is now at Removes use cache 2022-11-21T21:42:16.045Z [INFO]: Successfully cleaned up Git credentials 2022-11-21T21:42:16.045Z [INFO]: # Checking for Git submodules at: /codebuild/output/src067068951/src/lawhive.co.uk/.gitmodules 2022-11-21T21:42:16.055Z [INFO]: # Retrieving environment cache... 2022-11-21T21:42:16.091Z [WARNING]: ! Unable to write cache: {"code":"ERR_BAD_REQUEST","message":"Request failed with status code 404"})} 2022-11-21T21:42:16.091Z [INFO]: ---- Setting Up SSM Secrets ---- 2022-11-21T21:42:16.091Z [INFO]: SSM params {"Path":"/amplify/d3lq4oet4t4ooj/LH-885/performance/","WithDecryption":true} 2022-11-21T21:42:16.118Z [WARNING]: !Failed to set up process.env.secrets 2022-11-21T21:42:16.123Z [INFO]: # Defaulting to Node version 16 2022-11-21T21:42:25.583Z [INFO]: # Node version 16 is available for installation 2022-11-21T21:42:25.667Z [INFO]: # Installing Node version 16 2022-11-21T21:42:32.064Z [INFO]: # Now using Node version 16 2022-11-21T21:42:32.135Z [INFO]: No live updates for this build run 2022-11-21T21:42:32.137Z [INFO]: Setting NEXT_PRIVATE_STANDALONE=true to produce .next/standalone directory 2022-11-21T21:42:32.139Z [INFO]: # Retrieving cache... 2022-11-21T21:42:40.851Z [INFO]: # Extracting cache... 2022-11-21T21:42:47.670Z [INFO]: # Extraction completed 2022-11-21T21:42:47.811Z [INFO]: # Retrieved cache 2022-11-21T21:43:25.853Z [INFO]: BackendEnvironment name LH-885/performance for app d3lq4oet4t4ooj is invalid 2022-11-21T21:43:25.967Z [INFO]: Random environment name jacquelin generated 2022-11-21T21:43:26.822Z [INFO]: ## Starting Backend Build ## Checking for associated backend environment... ## No backend environment association found, continuing... ## Completed Backend Build 2022-11-21T21:43:26.824Z [INFO]: ## Starting Frontend Build 2022-11-21T21:43:26.824Z [INFO]: # Starting phase: preBuild # Executing command: yarn install 2022-11-21T21:43:27.100Z [INFO]: yarn install v1.22.0 2022-11-21T21:43:27.188Z [INFO]: [1/4] Resolving packages... 2022-11-21T21:43:27.534Z [INFO]: success Already up-to-date. 2022-11-21T21:43:27.538Z [INFO]: Done in 0.44s. 2022-11-21T21:43:27.547Z [INFO]: # Executing command: export NODE_OPTIONS="--max-old-space-size=5120" 2022-11-21T21:43:27.547Z [INFO]: # Completed phase: preBuild # Starting phase: build # Executing command: yarn run build 2022-11-21T21:43:27.762Z [INFO]: yarn run v1.22.0 2022-11-21T21:43:27.794Z [INFO]: $ next build 2022-11-21T21:43:28.181Z [INFO]: === Clearing Lawhive Data Build Cache 2022-11-21T21:43:28.210Z [INFO]: warn - No build cache found. Please configure build caching for faster rebuilds. Read more: https://nextjs.org/docs/messages/no-cache 2022-11-21T21:43:28.223Z [INFO]: Attention: Next.js now collects completely anonymous telemetry regarding usage. 2022-11-21T21:43:28.224Z [INFO]: This information is used to shape Next.js' roadmap and prioritize features. You can learn more, including how to opt-out if you'd not like to participate in this anonymous program, by visiting the following URL: https://nextjs.org/telemetry 2022-11-21T21:43:28.479Z [INFO]: info - Linting and checking validity of types... 2022-11-21T21:43:30.089Z [WARNING]: warn - The Next.js plugin was not detected in your ESLint configuration. See https://nextjs.org/docs/basic-features/eslint#migrating-existing-config 2022-11-21T21:43:30.135Z [WARNING]: Warning: React version not specified in eslint-plugin-react settings. See https://github.com/jsx-eslint/eslint-plugin-react#configuration . 2022-11-21T21:43:48.763Z [INFO]: info - Creating an optimized production build... 2022-11-21T21:43:49.055Z [INFO]: info - Disabled SWC as replacement for Babel because of custom Babel configuration ".babelrc.js" https://nextjs.org/docs/messages/swc-disabled 2022-11-21T21:43:49.564Z [INFO]: info - Using external babel configuration from /codebuild/output/src067068951/src/lawhive.co.uk/.babelrc.js 2022-11-21T21:44:52.045Z [WARNING]: Browserslist: caniuse-lite is outdated. Please run: npx browserslist@latest --update-db Why you should do it regularly: https://github.com/browserslist/browserslist#browsers-data-updating 2022-11-21T21:46:46.751Z [INFO]: Done in 199.00s. 2022-11-21T21:46:46.757Z [INFO]: # Completed phase: build ## Completed Frontend Build 2022-11-21T21:46:46.759Z [INFO]: ## Build completed successfully 2022-11-21T21:46:46.760Z [INFO]: # Starting caching... 2022-11-21T21:46:46.769Z [INFO]: # Creating cache artifact... 2022-11-21T21:47:00.963Z [INFO]: # Created cache artifact 2022-11-21T21:47:01.039Z [INFO]: # Uploading cache artifact... 2022-11-21T21:47:07.045Z [INFO]: # Uploaded cache artifact 2022-11-21T21:47:07.136Z [INFO]: # Caching completed 2022-11-21T21:47:07.139Z [INFO]: Setting NEXT_PRIVATE_STANDALONE=true to produce .next/standalone directory 2022-11-21T21:47:07.142Z [INFO]: # No custom headers found. 2022-11-21T21:47:07.146Z [ERROR]: !!! CustomerError: Standalone directory not found in /codebuild/output/src067068951/src/lawhive.co.uk/.next/standalone. Please enable output standalone on your next.config.js file or set NEXT_PRIVATE_STANDALONE=true. https://nextjs.org/docs/advanced-features/output-file-tracing#automatically-copying-traced-files 2022-11-21T21:47:07.146Z [INFO]: # Starting environment caching... 2022-11-21T21:47:07.146Z [INFO]: # Environment caching completed Terminating logging... ```

Additional information

No response

ghost commented 1 year ago

Hi @ShiftySlothe 👋🏽 Please try running a build locally to ensure that the standalone directory is being created:

export NEXT_PRIVATE_STANDALONE=true
npm run build # or yarn build

Also please ensure that you are using Next.js version 11 13 or 12.

ShiftySlothe commented 1 year ago

Hey @hloriana thanks for the response.

https://docs.aws.amazon.com/amplify/latest/userguide/ssr-Amplify-support.html makes it clear you support NextJS 13. Is this not the case?

I can also confirm the standalone repo is working locally.

EDIT:

I note that in the screenshot taken from the article about hosting Next13 "framework" is set.

image

However on my build settings I do not see this option.

image

EDIT 2: I've now created a new branch with just the start template you use on the article I linked above and it initially failed, having ran aws amplify update-branch setting --framework 'Next.js - SSR, it builds. I'm working through including my project files in the hope of identifying what's causing the error.

EDIT 3: I've successfully built the majority of the site now using Next13, I had to add pages incrementally, build and finally it would allow me to build all the pages! However, I now get this issue:

[ERROR]: !!! CustomerError: The size of the build output (483856351) exceeds the max allowed size of 230686720 bytes. Please reduce the size of your build output (/codebuild/output/src002007246/src/lawhive.co.uk/.next/standalone) and try again.

Is it possible to increase this figure?

ASI-Edward commented 1 year ago

I have met this issue too.

2022-11-23T03:21:23.355Z [ERROR]: !!! CustomerError: The size of the build output (425981801) exceeds the max allowed size of 230686720 bytes. Please reduce the size of your build output (/codebuild/output/src013718944/src/alphaswift/.next/standalone) and try again.

calavera commented 1 year ago

Hi, unfortunately there is a hard limit of 200MB for the bundle. Our current recommendation for now is to try to reduce the size by minimizing your files.

ShiftySlothe commented 1 year ago

Is there a plan to allow increases to that limit in the future and do you have an idea of timelines? We're minimising already however our site has a large number of pages and I doubt we can shave 50% off the bundle size.

calavera commented 1 year ago

this is a problem that we're looking into, but we don't have a timeline at the moment. Something that would help us is if you could add here the output of running these three commands:

du -h -d1 public du -h -d1 .next/static du -h -d1 .next/server

spreadpando commented 1 year ago

I ran into this same CustomerError: The size of the build output exceeds the max allowed size. Fortunately for this project I was able to minimize the contents of the public folder to sneak just under this limit. The workaround for me going forward with large asset-heavy next.js projects might be to host all my public assets on cloudfront/s3 and use the assetPrefix option in next.config.js to request static assets from cloudfront instead of the nextjs server.

This would be very in-line with next.js's edge-first philosophy. I'd be interested to see if the Amplify team could implement that kind of thing as part of the build process.

ovdahlberg commented 1 year ago

Hi,

We have the same issue here: !!! CustomerError: The size of the build output (676020993) exceeds the max allowed size of 230686720 bytes.

It worked before we upgraded to Web Compute with the same project.

I solved it temporarily by excluding paths in getStaticPaths and generating the pages on demand for some paths.

But that's not an ideal solution. The build output is limited to 220MB, meaning it's possible to statically build about 1 800 pages during the build process if each page is 125KB.

hect1c commented 1 year ago

Hey,

Just started to look to use AWS Amplify for my NextJS13 app. I've also ran into the build size issue. My NextJS is using SSR for all pages.

Errror !!! CustomerError: The size of the build output (238747580) exceeds the max allowed size of 230686720 bytes. Please reduce the size of your build output (/codebuild/output/src445488485/src/web-ui/.next/standalone) and try again.

I've attempted what was suggested by the docs and added the following to my build step in the .yml file

- allfiles=$(ls -al ./.next/standalone/**/*.js)
- npx esbuild $allfiles --minify --outdir=.next/standalone --platform=node --target=node16 --format=cjs --allow-overwrite

However this provides another error

# Executing command: allfiles=lrwxrwxrwx 1 root root 42 Dec  4 01:59 ./.next/standalone/node_modules/tippy.js -> .pnpm/tippy.js@6.3.7/node_modules/tippy.js
# Executing command: npx esbuild lrwxrwxrwx 1 root root 42 Dec  4 01:59 ./.next/standalone/node_modules/tippy.js -> .pnpm/tippy.js@6.3.7/node_modules/tippy.js --minify --outdir=.next/standalone --platform=node --target=node16 --format=cjs --allow-overwrite
✘ [ERROR] Invalid build flag: "->"  1 error

Not sure if this is related to the fact I'm using pnpm to install and manage my dependencies however I've checked esbuild support of pnpm and they appear to support symlinks https://github.com/evanw/esbuild/issues/310.

I've searched around for this error but have been unable to find anything.

Also here is an output of the three commands

╰─ du -h -d1 public
 13M    public/static
 13M    public

 ╰─ du -h -d1 .next/static
 72K    .next/static/css
 11M    .next/static/chunks
 20K    .next/static/ilatlR2YlvJYNcYiWmMmj
 11M    .next/static

 ╰─ du -h -d1 .next/server
6.6M    .next/server/chunks
 22M    .next/server/pages
644K    .next/server/src
 30M    .next/server

Will see if there's anything I can do to reduce it but any feedback or direction would be greatly appreciated.

treewhopper commented 1 year ago

Hey,

Just started to look to use AWS Amplify for my NextJS13 app. I've also ran into the build size issue. My NextJS is using SSR for all pages.

Errror !!! CustomerError: The size of the build output (238747580) exceeds the max allowed size of 230686720 bytes. Please reduce the size of your build output (/codebuild/output/src445488485/src/web-ui/.next/standalone) and try again.

I've attempted what was suggested by the docs and added the following to my build step in the .yml file

- allfiles=$(ls -al ./.next/standalone/**/*.js)
- npx esbuild $allfiles --minify --outdir=.next/standalone --platform=node --target=node16 --format=cjs --allow-overwrite

However this provides another error

# Executing command: allfiles=lrwxrwxrwx 1 root root 42 Dec  4 01:59 ./.next/standalone/node_modules/tippy.js -> .pnpm/tippy.js@6.3.7/node_modules/tippy.js
# Executing command: npx esbuild lrwxrwxrwx 1 root root 42 Dec  4 01:59 ./.next/standalone/node_modules/tippy.js -> .pnpm/tippy.js@6.3.7/node_modules/tippy.js --minify --outdir=.next/standalone --platform=node --target=node16 --format=cjs --allow-overwrite
✘ [ERROR] Invalid build flag: "->"  1 error

Not sure if this is related to the fact I'm using pnpm to install and manage my dependencies however I've checked esbuild support of pnpm and they appear to support symlinks evanw/esbuild#310.

I've searched around for this error but have been unable to find anything.

Also here is an output of the three commands

╰─ du -h -d1 public
 13M    public/static
 13M    public

 ╰─ du -h -d1 .next/static
 72K    .next/static/css
 11M    .next/static/chunks
 20K    .next/static/ilatlR2YlvJYNcYiWmMmj
 11M    .next/static

 ╰─ du -h -d1 .next/server
6.6M    .next/server/chunks
 22M    .next/server/pages
644K    .next/server/src
 30M    .next/server

Will see if there's anything I can do to reduce it but any feedback or direction would be greatly appreciated.

It's over a month later, but I found that:

  1. The ls -al did not like the globstar (**) and included directories and file path symbols such as the symlinks you mentioned.
  2. There were too many files for me to run in a single pass of esbuild on the default Amplify build container.

If you create a minify.sh bash script and add it to your build steps in amplify.yml, this worked for reducing my bundle size:

#!/bin/bash
cd .next/standalone/node_modules
for d in $(find . -maxdepth 1 -type d); do
  npx --yes esbuild $(find $d -type f -name '*.js') --minify --outdir=$d --platform=node --target=node16 --format=cjs --allow-overwrite
done
cd -

Unfortunately, even after meeting the size requirements, WEB_COMPUTE is currently having a deployment bundling issue that is causing deployments to fail silently #3213.

danwsc09 commented 1 year ago

It's over a month later, but I found that:

  1. The ls -al did not like the globstar (**) and included directories and file path symbols such as the symlinks you mentioned.
  2. There were too many files for me to run in a single pass of esbuild on the default Amplify build container.

If you create a minify.sh bash script and add it to your build steps in amplify.yml, this worked for reducing my bundle size:

#!/bin/bash
cd .next/standalone/node_modules
for d in $(find . -maxdepth 1 -type d); do
  npx --yes esbuild $(find $d -type f -name '*.js') --minify --outdir=$d --platform=node --target=node16 --format=cjs --allow-overwrite
done
cd -

Unfortunately, even after meeting the size requirements, WEB_COMPUTE is currently having a deployment bundling issue that is causing deployments to fail silently #3213.

I've run this script and I can't seem to get my output size down any further. I've also tried running esbuild on .next/standalone/.next but that didn't help very much.

cd .next/standalone/node_modules
for d in $(find . -maxdepth 1 -type d); do
  npx --yes esbuild $(find $d -type f -name '*.js') --minify --outdir=$d --platform=node --target=node16 --format=cjs --allow-overwrite
done
cd -
cd .next/standalone/.next
for d in $(find . -maxdepth 1 -type d); do
  npx --yes esbuild $(find $d -type f -name '*.js') --minify --outdir=$d --platform=node --target=node16 --format=cjs --allow-overwrite
done
cd -

Running esbuild helped a bit as I've gotten the bundle size from 240572524 to 234879691 but there still remains ~5 MB.

CustomerError: The size of the build output (234879691) exceeds the max allowed size of 230686720 bytes. Please reduce the size of your build output (/codebuild/output/src129382/src/my-project/.next/standalone) and try again

Any help reducing this bundle size would be greatly appreciated.

pcolmer commented 1 year ago

@calavera

this is a problem that we're looking into, but we don't have a timeline at the moment. Something that would help us is if you could add here the output of running these three commands:

$ du -h -d1 public
604K    public/posts
48K     public/images
3.1M    public

$ du -h -d1 .next/static
34M     .next/static/chunks
24K     .next/static/ICQyTuaWXQ1OObGk8yzsN
152K    .next/static/css
34M     .next/static

$ du -h -d1 .next/server
1.7M    .next/server/chunks
21M     .next/server/pages
23M     .next/server

Running the minify script seemed to do something ... but the output from du didn't change one bit so I'm not sure if anything was achieved :(

Updating to add that when getting Amplify to build my application, the minify script reduced the build output from 463721822 to 450400919 ... so that didn't really help much :(

borisyordanov commented 1 year ago

FYI standalone dir errors can appear also when using a monorepo - https://github.com/aws-amplify/amplify-hosting/issues/3179

agrattan0820 commented 1 year ago

Noting here that I am also a victim of the Amplify build limit for a production application we're looking to upgrade to Next.js 13 🥲 !!! CustomerError: The size of the build output (344395275) exceeds the max allowed size of 230686720 bytes. Please reduce the size of your build output (/codebuild/output/src888343114/src/alcohol-101-plus/services/app/.next/standalone) and try again

And this is after using ESBuild following this StackOverflow post

mattr5341989 commented 1 year ago

Has anyone found a solution to the original problem. I also am getting an error for not having a standalone directory.

LiamAshdown commented 1 year ago

I get the same issue, and it's a road blocker for me.

LiamAshdown commented 1 year ago

du -h -d1 .next/server

I'm stuck too, and would need to find an alternative hosting provider due to this limitation.

hect1c commented 1 year ago

I was having issues as well similar to those above however we couldn't wait and really wanted to reap the benefits of serverless, edge and nextjs 13. Luckily we found another solution that also deploys to AWS services (lambda, apigateway, etc) and has been working really well in our tests so far: https://sst.dev/. Would recommend for those that want to still utilize AWS and need a working solution now.

GA-20 commented 1 year ago

I solved the error "CustomerError: Standalone directory not found" doing the following:

const nextConfig = {
  reactStrictMode: true,
  output: 'standalone',
  images: {
    unoptimized: true,
  }
}

module.exports = nextConfig

Step 3: Update the build script in the package.json file to include the next export command, which generates the static files for the app:

"scripts": {
  "dev": "next dev",
  "build": "next build && next export",
  "start": "next start",
  "lint": "next lint"
}

Step 4: Change the output directory to out. This is where the generated static files will be stored. Here's an example of what your deployment configuration might look like:

version: 1
frontend:
  phases:
    preBuild:
      commands:
        - yarn install
    build:
      commands:
        - yarn run build
  artifacts:
    baseDirectory: out
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**

I hope it works for you.

talktothelaw commented 1 year ago

I still having this issue

sawakaga commented 1 year ago

NextJS 13 error still occurs.

amplify.yml has very straightforward configuration

version: 1
backend:
  phases:
    build:
      commands:
        - '# Execute Amplify CLI with the helper script'
        - amplifyPush --simple
frontend:
  phases:
    preBuild:
      commands:
        - yarn install
    build:
      commands:
        - yarn build
  artifacts:
    baseDirectory: .next
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*
du -s .next/standalone/ 

159720 .next/standalone/

Standalone folder approximately 80 mbs

Screenshot 2023-05-03 at 15 33 05

As you see in the screenshot. On Amplify build phase it still gives error for exceeding the build output size.

kachkaev commented 1 year ago

@sawakaga have you managed to find a solution? Seeing the same on Amplify today. Running du -sh .next/standalone locally outputs 109MB.

rafaelvicio commented 1 year ago

same problem here

kachkaev commented 1 year ago

@rafaelvicio if you use pnpm, can you try adding pnpm config set node-linker hoisted before pnpm install? Seems like this solved the problem for us.

hoop71 commented 1 year ago

@kachkaev Thank you.

@rafaelvicio if you use pnpm, can you try adding pnpm config set node-linker hoisted before pnpm install? Seems like this solved the problem for us.

This worked for me by placing these commands into an .npmrc file. Sub-optimal but it unblocked the build.

!!! CustomerError: The size of the build output (xxx) exceeds the max allowed size of xxx bytes. Error

sawakaga commented 1 year ago

@sawakaga have you managed to find a solution? Seeing the same on Amplify today. Running du -sh .next/standalone locally outputs 109MB.

Problem still occurs unfortunately. Currently we dont host application on amplify

arpit-exp commented 1 year ago

Just wanted to update you all that I successfully migrated our company's Next.js 11 project to Next.js 13. However, I encountered an issue with the bundle size, which was over 900MB. I reached out to AWS support, and after some discussions and various attempts to fix it, we concluded that we need to wait for the issue to be resolved as they are currently working on it.

In case you're interested, here's the documentation I found regarding a potential solution using the command placement in the amplify.yml file after the build step: Amplify Troubleshooting SSR Deployment

mayur-2345 commented 1 year ago

facing same issue

calavera commented 1 year ago

Hey everyone 👋

We just released some changes to improve how we bundle Next.js apps. These changes don't need the standalone directory anymore. That should help eliminate the problem of reaching the 200MB bundle limit. These changes require a new deploy in your application to take effect. The expected outcome is that your app bundles should be smaller, and not bump into the size limit.

If you have comments or questions don't hesitate to reach us via AWS support.

github-actions[bot] commented 1 year ago

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.

github-actions[bot] commented 1 year ago

This issue has been automatically locked.