[NextJS][ISR] intermittent 503 errors for ISR pages

andrewgadziksonos commented 3 years ago

Before opening, please confirm:

[X] I have checked to see if my question is addressed in the FAQ.
[X] I have searched for duplicate or closed issues.
[X] I have read the guide for submitting bug reports.
[X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

App Id

dh2ndod2t9wbt

Region

us-east-1

Amplify Console feature

Custom builds, Performance

Describe the bug

SSR Lambda is configured with a 10 second timeout which is causing ISR routes to return back an intermittent 503 error based on cold starts, data fetching, etc.

Expected behavior

SSR Lambdas should be configured with a 30 second timeout, or allow for a configuration override

Reproduction steps

Create a NextJS app with an ISR route that contains a few data fetching calls, fallback = "blocking"
Deploy to Amplify
Request ISR page that was no pre-rendered during the build
Notice 503

Build Settings

version: 1
frontend:
  phases:
    preBuild:
      commands:
        - yarn install
    build:
      commands:
        - yarn run build
  artifacts:
    baseDirectory: .next
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*
      - .next/cache/**/*

Additional information

I have to manually go into the console and change the timeout value from 10 to 30 between every deploy

quinnturner commented 3 years ago

I had to update the IAM policy of the Lambda manually to include:

        {
            "Effect": "Allow",
            "Resource": "arn:aws:sqs:us-east-1:253925498289:<SQS-ID>.fifo",
            "Action": [
                "sqs:SendMessage"
            ]
        }

Does your Lambda have the correct permissions?

I am in the process of writing a GitHub issue dedicated to this problem.

andrewgadziksonos commented 3 years ago

I had to update the IAM policy of the Lambda manually to include:
        {
            "Effect": "Allow",
            "Resource": "arn:aws:sqs:us-east-1:253925498289:<SQS-ID>.fifo",
            "Action": [
                "sqs:SendMessage"
            ]
        }
Does your Lambda have the correct permissions?

I am in the process of writing a GitHub issue dedicated to this problem.

Yeah I can confirm that I have the proper IAM policy.

Here's the CloudWatch logs

Athena96 commented 3 years ago

yes, if you create an app initially not using ISR, then you add an ISR page, then you will need to manually update the IAM policy on the lambda.

drmzio commented 3 years ago

Also have the exact same issue. My API route averages 18-20s on local, but when deploying to Amplify, it times-out at 10s.

Heres my StackOverflow post: https://stackoverflow.com/questions/68957437/how-to-increase-timeout-of-amplify-nextjs-ssr-function-from-10s-to-30s

nicksemansonos commented 3 years ago

We're having a similar issue with a Next.js app. The API routes we configured often can take more than 10 seconds so we'd love to be able to configure the default timeout on the auto-generated Lambdas. It's possible to change it on the Lambda manually, but it will get overwritten next time the project builds.

swaminator commented 3 years ago

@nicksemansonos @drmzio we have updated the default timeout to 30sec. Please let us know if you're still having issues.

andrewgadziksonos commented 3 years ago

@nicksemansonos @drmzio we have updated the default timeout to 30sec. Please let us know if you're still having issues.

This is amazing @swaminator, thank you!

We've had a redploy in the last 9 hours and I'm still noticing that our lambdas are configured with a 10 sec timeout. Anything special we need to do in order to get the new values?

konyu commented 2 years ago

@nicksemansonos @drmzio we have updated the default timeout to 30sec. Please let us know if you're still having issues.

I still have this problem in my ISR next.js app. What should I do any change my environment?

Thanks.

ChrisAtVirtueraDev commented 2 years ago

Hello! I'm also facing the issue of getting 503 pages. I have a page, where getStaticPaths returns { path: [], fallback: true} and getStaticProps returns {..., revalidate: 900}. First I got the 503 immediately after accessing that page. Which was resolved by increasing the lambda timeouts to 30sec (so i guess the lambda function responsible for the fallback prerendering timed out). But the 503 occurred again on the same page later, when the regeneration was triggered. This time it was the lacking sqs permissions of the lambda functions. I'm not sure though how I am supposed to resolve this issue in a way that persists over redeploys. Am I supposed to completly recreate my amplify project? Because from the Amplify UI perspective, there is nothing different.

yogeshydvme commented 2 years ago

In ISR lambda we are still seeing 10 sec as default timeout, whereas other lambda have 30sec. Please help it is blocking us to adopt amplify

jnastaskin commented 2 years ago

I had to update the IAM policy of the Lambda manually to include:
        {
            "Effect": "Allow",
            "Resource": "arn:aws:sqs:us-east-1:253925498289:<SQS-ID>.fifo",
            "Action": [
                "sqs:SendMessage"
            ]
        }
Does your Lambda have the correct permissions?

I am in the process of writing a GitHub issue dedicated to this problem.

I'm also having this 503 issue with a deployed page using getStaticPaths/Props.. @quinnturner when you say lambda which lambda are you referring to? After deploying I can see these lambdas with what I assumed was their IAM role name:

however, i couldn't find those roles in IAM and so wasn't sure where to update the policy, the closest match was these:

do I need to manually create these roles or am i just looking in the wrong place?

jnastaskin commented 2 years ago

Update: found how to get those role names (had to change the location in lambda console to see the lambdas then get their role names from there), checked the permissions and everything is set but still seeing 503 errors on deployed pages using getStaticPaths/Props

alacret commented 2 years ago

UPDATE: This was happening to me on next@12.0.9, After testing many many things, I just downgraded the version to next@12.0.8 and it started working normally.

ocbrollingpaper commented 2 years ago

UPDATE: This was happening to me on next@12.0.9, After testing many many things, I just downgraded the version to next@12.0.8 and it started working normally.

same happening to latest version, had to downgrade, any update on this?

ssomlk commented 1 year ago

I upgraded from NextJs 11.1.2 to Next 12.3. I cannot see the Image. Gives the 503 error

https://ssomlk-bugfixes3.d31jx5fml0tv9b.amplifyapp.com/_next/image?url=%2F_next%2Fstatic%2Fimage%2Fpublic%2FIIT-2.717a6c886d0cf6b638cf3c0eb928d65f.svg&w=384&q=75

But the weird thing is I am not using next/image but a normal HTML img tag

Tried downgrading to next@12.0.8 but still no luck

aws-amplify / amplify-hosting