search

aws-amplify / amplify-hosting

AWS Amplify Hosting provides a Git-based workflow for deploying and hosting fullstack serverless web applications.
https://aws.amazon.com/amplify/hosting/
Apache License 2.0
461 stars 116 forks source link

[build] Node 16 Frontend NextJS build fails with error: Starting SSR Build... Terminating logging... #2412

Closed opqpop closed 3 years ago

opqpop commented 3 years ago

Before opening, please confirm:

App Id

d187t7owzjiq4m

Region

us-east-1

Amplify Console feature

Backend builds, Build settings, Custom builds

Describe the bug

Please help escalate this prod breaking bug for us that won't let us deploy prod fixes and new features.

I have a NextJS frontend build setup on Amplify that has been working for months up to 11/16/21 (Build 98: https://console.aws.amazon.com/amplify/home?region=us-east-1#/d187t7owzjiq4m/main/98). We then pushed on 11/24/21 some minor prod changes and couldn't deploy it due to build failing (Build 99: https://console.aws.amazon.com/amplify/home?region=us-east-1#/d187t7owzjiq4m/main/99).

The error logs aren't useful, it just says "Terminating logging..." after "Starting SSR Build...", and we are stumped on how to debug what's causing the build to break all of a sudden. All our builds since then fail because of this issue.

Code changes are minor and mostly just javascript refactoring, so I think this this is some client upgrade / versioning / amplify issue that happened between 11/16 - 11/24.

Unhelpful logs that just says "Terminating logging..." after "Starting SSR Build...

2021-11-25T05:41:09.928Z [INFO]: Ξ»  (Server)  server-side renders at runtime (uses getInitialProps or getServerSideProps)
                                 β—‹  (Static)  automatically rendered as static HTML (uses no initial props)
                                 ●  (SSG)     automatically generated as static HTML + JSON (uses getStaticProps)
                                 (ISR)     incremental static regeneration (uses revalidate in getStaticProps)
2021-11-25T05:41:10.136Z [INFO]: Starting SSR Build...
Terminating logging...

Full log: log (5).txt

Expected behavior

Logging output should show exact error instead of "Terminating logging..."

Reproduction steps

All our builds are now failing since the last successful 11/16/21. I think I just need to know what the error is at the build step and I can probably figure out the fix. Unfortunately the amplify console logging is not outputting the error.

Build Settings

version: 1
applications:
  - backend:
      phases:
        build:
          commands:
            - amplifyPush --simple
    frontend:
      phases:
        preBuild:
          commands:
            - nvm install && nvm use
            - npm install
        build:
          commands:
            - npm run build
      artifacts:
        baseDirectory: .next
        files:
          - "**/*"
      cache:
        paths:
          - node_modules/**/*
    appRoot: frontend

Additional information

In the past, I've seen similar issues, and a fix for me was I forgot some env variables that was needed for building (such as FontAwesome env variables to download fontawesome pro package). I don't think this is relevant this time around though, since I haven't made any changes that would require new env variables during the build for installing new packages.

ciceropablo commented 3 years ago

I have the same problem. Any news about it?

wellguimaraes commented 3 years ago

Same problem here 😒

wellguimaraes commented 3 years ago

Tried even a very basic Next app with SSR and the problem remains

che-ri commented 3 years ago

same problem 😭😭

zibra commented 3 years ago

+1

wellguimaraes commented 3 years ago

Apparently, the problem is happening on specific regions (us-east-1, us-east-2 and us-west-2 at least). On us-west-1 the build step succeeds.

wellguimaraes commented 3 years ago

Errrr, not anymore... Failing everywhere now...

mahmoudmiz commented 3 years ago

same problem as well!

Rafcin commented 3 years ago

@opqpop @wellguimaraes @che-ri @ciceropablo @zibra

Can you add your package json, amplify build yml, next config, and what type of repo you are using if you're allowed to? This seems to be a common issue.

I'm currently deploying a NextJS monorepo with NX and according to the amplify team it's a problem with npx serverless just failing.

Rafcin commented 3 years ago

Also stay away from deploying NextJs versions 12.0.1, 12.0.2 and 12.0.3 due to rust compiler issues. 12.0.4 should be stable and 12.0.5-canary.0 is also stable.

opqpop commented 3 years ago

@opqpop @wellguimaraes @che-ri @ciceropablo @zibra

Can you add your package json, amplify build yml, next config, and what type of repo you are using if you're allowed to? This seems to be a common issue.

@Rafcin

package.json: https://gist.github.com/opqpop/a53282aaab476d56ccd5589d25541c9e package-lock.json: https://gist.github.com/opqpop/27fa75baf64723b41b46dfcc3668080b next.config.js: https://gist.github.com/opqpop/333456b74f1daf639a0e38aaf0d8d509 type of repo: github connected on amplify

amplify.yml

version: 1
applications:
  - backend:
      phases:
        build:
          commands:
            - amplifyPush --simple
    frontend:
      phases:
        preBuild:
          commands:
            - nvm install && nvm use
            - npm install
        build:
          commands:
            - npm run build
      artifacts:
        baseDirectory: .next
        files:
          - "**/*"
      cache:
        paths:
          - node_modules/**/*
    appRoot: frontend

npm list

(venv) winston@Winstons-MacBook-Pro ~/f/frontend (bowza11)> npm list                                                                                                                                                                                                  11:15:50
directory_react_theme@1.1.0 /Users/winston/flueth/frontend
+-- @babel/cli@7.15.7
+-- @babel/core@7.15.5
+-- @babel/polyfill@7.12.1
+-- @babel/preset-env@7.15.6
+-- @babel/preset-react@7.14.5
+-- @fortawesome/fontawesome-free@5.15.4
+-- @fortawesome/fontawesome-pro@5.15.4
+-- @fortawesome/fontawesome-svg-core@1.2.36
+-- @fortawesome/free-solid-svg-icons@5.15.4
+-- @fortawesome/react-fontawesome@0.1.15
+-- @types/google-maps@3.2.3
+-- aws-amplify@4.2.10
+-- babel-loader@8.2.2
+-- babel-plugin-macros@3.1.0
+-- babel-plugin-relay@12.0.0
+-- babel-preset-react-app@10.0.0
+-- detect-file@1.0.0
+-- google-maps-react@2.0.6
+-- graphql@15.6.0
+-- hsl-to-hex@1.0.0
+-- leaflet@1.7.1
+-- lodash.clonedeep@4.5.0
+-- moment-timezone@0.5.33
+-- next-compose-plugins@2.2.1
+-- next-google-fonts@1.2.1
+-- next-react-svg@1.1.3
+-- next@11.0.1
+-- nouislider@15.4.0
+-- nprogress@0.2.0
+-- object-fit-images@3.2.4
+-- prettier@2.4.1
+-- react-dates@21.8.0
+-- react-dom@17.0.2
+-- react-dropzone@11.4.0
+-- react-error-boundary@3.1.3
+-- react-ga@3.3.0
+-- react-hook-form@7.15.4
+-- react-id-swiper@4.0.0
+-- react-image-lightbox@5.1.4
+-- react-leaflet@3.2.1
+-- react-loading-overlay@1.0.1
+-- react-outside-click-handler@1.3.0
+-- react-places-autocomplete@7.3.0
+-- react-relay@12.0.0
+-- react-scroll@1.8.4
+-- react-select@4.3.1
+-- react-syntax-highlighter@15.4.4
+-- react@17.0.2
+-- reactstrap@8.10.0
+-- relay-compiler@12.0.0
+-- relay-config@12.0.0
+-- relay-runtime@12.0.0
+-- sass@1.42.0
+-- styled-components@5.3.1
+-- swiper@6.8.4
`-- wnumb@1.2.0

Note I've always used build settings NextJS version set to 10 and it always worked, but I've been using Next 11.0.1 so not sure why it worked. Anyways, I also tried setting NextJS version to latest and got same error.

Let me know if you need anything else. Thanks for the help!

Rafcin commented 3 years ago

@hloriana This issue and the monorepo issue might be sharing the same build problem. https://github.com/aws-amplify/amplify-console/issues/2377

Rafcin commented 3 years ago

@opqpop I'm looking at your build yml. It looks good minus the nvm part. I don't think you need nvm unless you need to set a specific version, and even then in the console, you can add a package override.

Have you tried redeploying on a new app just as a test?

Odd, I've tested from Next 10 to 12 and I've had failures on all of them. I should note before I switch to a monorepo I successfully deployed Next 12 without any issues the afternoon it came out.

I think it's a serverless issue, but without the build logs / a staff member to read them, we won't really know :( I'll see if I can find out how to print the debug logs.

Also, I would attach your region ID's to the above posts so when a service member checks it out they can look at the logs immediately for your deployment.

swaminator commented 3 years ago

@ciceropablo @zibra @wellguimaraes @opqpop @Rafcin we are looking into this now with priority.

Rafcin commented 3 years ago

Adding some related issues that involve the SSR deployment bug. https://github.com/vercel/next.js/issues/31555 - Monorepo | NextJS https://github.com/bmvantunes/aws-amplify-deploy-test-nx/issues/1 - This guy built his app and managed to deploy it somehow. (I think he's making a video on this) https://github.com/aws-amplify/amplify-console/issues/2377 - Same build issues https://github.com/aws-amplify/amplify-console/issues/2387 - Looked like an amplify pkg error

Rafcin commented 3 years ago

@swaminator Thank you! On Discord Kgnanask mentioned at least for my case this was an issue with serverless. npx serverless just terminated without any response. If you want to see it my dist id is d2wuxof40gra3l. I deploy it similarly to how some of the above are deployed.

Will this be the best place to check for updates? I opened a support ticket although they don't have any updates and our messages are very delayed.

From what's known now, is this happening to single apps and monorepos? I need to have a deployment out within the next few days, and I haven't been able to solve this and the cloudfront header issue since the end of September. So if it's an issue that's easily solvable for single apps will it be worth it for me to move back to a single app or should I stay with my monorepo project. At this point I just need to deploy something but moving back to a single app will require me to restructure and redeploy everything so I need to know what's going on with this issue asap!

victorccccc commented 3 years ago

Hi @ciceropablo @zibra @wellguimaraes, can we also have your AppId please?

ciceropablo commented 3 years ago

Hi @victorccccc! My application id is dekdvlzg04mz2.

j-mastr commented 3 years ago

Hi, I'm experiencing the same issue, even by rolling back to an previously successful build (~5 days ago)

My app ID is: dc5j734acopyd Running on eu-central-1, nextjs version is 11.0.1

swaminator commented 3 years ago

Tried even a very basic Next app with SSR and the problem remains

@wellguimaraes I was able to deploy a basic Next app with SSR without issues: https://github.com/swaminator/create-next-app. Does anyone have a repo they can help with that is failing?

Rafcin commented 3 years ago

@swaminator @victorccccc I'll send you an invite to my repo, I'm still stuck in the failing state. (Under dev)

j-mastr commented 3 years ago

@swaminator @victorccccc same for my repo

ferdingler commented 3 years ago

Can you confirm if you are using node v16? It seems to be happening only with that version.

swaminator commented 3 years ago

@Rafcin @j-mastr we were able to reproduce the error with Node 16. In the interim, while we figure out why this is happening please downgrade to Node 12.

Rafcin commented 3 years ago

I'll test node 14 again. Node 14 also failed in the past but i'll double check.

wellguimaraes commented 3 years ago

@swaminator Thanks for checking this one.

Just tried with this project and had no success: https://github.com/wellguimaraes/nextjs-basic-ssr

––––

I've just noticed the comment about node version. I'll try with v12

Rafcin commented 3 years ago

@swaminator I switched to 14 and the same build issue occured.

ferdingler commented 3 years ago

To try it with node 14, you can add this to your amplify.yml in the build section. Our integration tests run with node 14 and they are all passing, this issue shouldn't occur with node14. 

nvm install 14
nvm use 14

We are still root causing what is going on with node 16.

Rafcin commented 3 years ago

I just tested v12.22.7 and it threw the same issue. (Build 49)

Rafcin commented 3 years ago

@ferdingler v14 failed as well (Build 50), same termination error.

ferdingler commented 3 years ago

@Rafcin we looked at your build logs and it seems to be unrelated to this. Are you using the NX monorepo framework? We have seen issue with some advanced features of that framework because of the way it wraps the build artifacts. We will track this as well, but it doesn't seem related to this particular issue of node 16. https://github.com/aws-amplify/amplify-console/issues/2377

ferdingler commented 3 years ago

We believe the issue with node 16 is related to npx now expecting user confirmation which causes the build to exit immediately: https://github.com/npm/cli/issues/2226. We use npx for build commands on SSR apps. We are testing to confirm this theory, but in the meantime pinning your node version to 14 should be a safe workaround.

swaminator commented 3 years ago

@wellguimaraes I was able to get your basic repo to succeed using Node 14: https://github.com/swaminator/nextjs-basic-ssr/blob/main/amplify.yml

Rafcin commented 3 years ago

@ferdingler Yep NX. I assumed the termination issue might have been related. I really had no way of knowing because I can't see the full extent of the logs.

I have an issue open https://github.com/aws-amplify/amplify-console/issues/2377 but haven't found any solution to the issue. It's making me really nervous because the AWS ticket and all the issues haven't yielded a solution and I'm running out of options.

wellguimaraes commented 3 years ago

@wellguimaraes I was able to get your basic repo to succeed using Node 14: https://github.com/swaminator/nextjs-basic-ssr/blob/main/amplify.yml

@swaminator Yep! Got things to work here with Node14. Thanks!

j-mastr commented 3 years ago

Just tested v14.18.1 and it seems to work without issues for me πŸŽ‰ Thanks!

swaminator commented 3 years ago

@Rafcin we were able to reproduce your issue - we will respond to you in the #2377 ticket so that we don't conflate many things here. This issue is specifically with Node 16.

opqpop commented 3 years ago

To try it with node 14, you can add this to your amplify.yml in the build section. Our integration tests run with node 14 and they are all passing, this issue shouldn't occur with node14.

nvm install 14
nvm use 14

We are still root causing what is going on with node 16.

confirming this fixed it for us! Thanks for the quick response. Looking forward to learning about the root cause

my node version locally: 

(venv) winston@Winstons-MacBook-Pro ~/f/frontend (bowza12)> node -v     
v16.4.2
ferdingler commented 3 years ago

We rolled out a fix for node 16. Feel free to switch back to it if needed. Apologies the inconvenience.

github-actions[bot] commented 2 years ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.