Credentials may not be refreshed on long build.

ymsrk commented 2 years ago

Before opening, please confirm:

[X] I have checked to see if my question is addressed in the FAQ.
[X] I have searched for duplicate or closed issues.
[X] I have read the guide for submitting bug reports.
[X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

App Id

d13pw8k8p2rhkg

Region

ap-northeast-1

Amplify Hosting feature

Build settings

Describe the bug

I am trying to deploy my nextjs app on aws through CodeCommit repository. But the build is failing and giving me below pasted message.

"ExpiredToken: The security token included in the request is expired"

Expected behavior

No exceptions, automatic token refresh if operation takes long.

Reproduction steps

This occurs after a long build time.
The probability of this occurring increases when Generating static pages exceeds 6000 pages.

Build Settings

version: 1
frontend:
  phases:
    preBuild:
      commands:
        - amplifyPush --simple
        - rm -rf node_modules
        - yarn install
    build:
      commands:
        - yarn run build
  artifacts:
    baseDirectory: .next
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*

Additional information

2022-05-13T11:39:57.693Z [INFO]: info  - Generating static pages (0/6189)
2022-05-13T11:40:58.011Z [INFO]: info  - Generating static pages (238/6189)
2022-05-13T11:41:58.353Z [INFO]: info  - Generating static pages (442/6189)
2022-05-13T11:42:58.370Z [INFO]: info  - Generating static pages (751/6189)
2022-05-13T11:43:58.514Z [INFO]: info  - Generating static pages (980/6189)
2022-05-13T11:44:58.553Z [INFO]: info  - Generating static pages (1408/6189)
2022-05-13T11:45:16.698Z [INFO]: info  - Generating static pages (1547/6189)
2022-05-13T11:46:16.947Z [INFO]: info  - Generating static pages (1938/6189)
2022-05-13T11:47:17.183Z [INFO]: info  - Generating static pages (2256/6189)
2022-05-13T11:48:17.386Z [INFO]: info  - Generating static pages (2486/6189)
2022-05-13T11:49:17.494Z [INFO]: info  - Generating static pages (2692/6189)
2022-05-13T11:50:17.641Z [INFO]: info  - Generating static pages (2919/6189)
2022-05-13T11:51:00.133Z [INFO]: info  - Generating static pages (3094/6189)
2022-05-13T11:52:00.264Z [INFO]: info  - Generating static pages (3349/6189)
2022-05-13T11:53:00.329Z [INFO]: info  - Generating static pages (3594/6189)
2022-05-13T11:54:00.581Z [INFO]: info  - Generating static pages (3805/6189)
2022-05-13T11:55:00.636Z [INFO]: info  - Generating static pages (3998/6189)
2022-05-13T11:56:00.690Z [INFO]: info  - Generating static pages (4329/6189)
2022-05-13T11:57:00.806Z [INFO]: info  - Generating static pages (4613/6189)
2022-05-13T11:57:11.404Z [INFO]: info  - Generating static pages (4641/6189)
2022-05-13T11:58:11.519Z [INFO]: info  - Generating static pages (4828/6189)
2022-05-13T11:59:11.553Z [INFO]: info  - Generating static pages (5058/6189)
2022-05-13T12:00:11.829Z [INFO]: info  - Generating static pages (5300/6189)
2022-05-13T12:01:12.448Z [INFO]: info  - Generating static pages (5539/6189)
2022-05-13T12:02:12.681Z [INFO]: info  - Generating static pages (5766/6189)
2022-05-13T12:03:12.821Z [INFO]: info  - Generating static pages (5962/6189)
2022-05-13T12:04:12.891Z [INFO]: info  - Generating static pages (6177/6189)
2022-05-13T12:04:15.921Z [INFO]: info  - Generating static pages (6189/6189)
2022-05-13T12:04:15.934Z [INFO]: info  - Finalizing page optimization...
2022-05-13T12:04:17.330Z [INFO]: 
2022-05-13T12:04:17.396Z [INFO]: Page                                                       Size     First Load JS
                                 ┌ ● / (687 ms)                                             1.21 kB         166 kB
                                 ├   /_app                                                  0 B            69.9 kB

...
2022-05-13T12:04:17.708Z [INFO]: Done in 1503.58s.
2022-05-13T12:04:17.713Z [INFO]: Starting SSR Build...
2022-05-13T12:35:40.334Z [ERROR]: ExpiredToken: The security token included in the request is expired
                                  at Request.extractError (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/protocol/rest_xml.js:53:29)
                                  at Request.callListeners (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
                                  at Request.emit (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
                                  at Request.emit (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/request.js:686:14)
                                  at Request.transition (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/request.js:22:10)
                                  at AcceptorStateMachine.runTo (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/state_machine.js:14:12)
                                  at /root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/state_machine.js:26:10
                                  at Request.<anonymous> (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/request.js:38:9)
                                  at Request.<anonymous> (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/request.js:688:12)
                                  at Request.callListeners (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
                                  at Request.emit (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
                                  at Request.emit (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/request.js:686:14)
                                  at Request.transition (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/request.js:22:10)
                                  at AcceptorStateMachine.runTo (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/state_machine.js:14:12)
                                  at /root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/state_machine.js:26:10
                                  at Request.<anonymous> (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/request.js:38:9)
                                  at Request.<anonymous> (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/request.js:688:12)
                                  at Request.callListeners (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
                                  at callNextListener (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/sequential_executor.js:96:12)
                                  at IncomingMessage.onEnd (/root/.//node_modules/@sls-next/cloudfront/node_modules/aws-sdk/lib/event_listeners.js:335:13)
                                  at IncomingMessage.emit (node:events:402:35)
                                  at IncomingMessage.emit (node:domain:475:12)
                                  at endReadableNT (node:internal/streams/readable:1343:12)
                                  at processTicksAndRejections (node:internal/process/task_queues:83:21) {
                                  code: 'ExpiredToken',
                                  time: 2022-05-13T12:35:40.124Z,
                                  requestId: '826a7bc1-8d34-4081-b410-1711c35d70db',
                                  statusCode: 403,
                                  retryable: true
                                  }
Terminating logging...

ghost commented 2 years ago

Hi @ymsrk 👋🏽 thanks for raising this issue. We are tracking this bug for prioritization. We will update this issue once we have more information

ghost commented 2 years ago

We have added this bug to our backlog for prioritization and the team is investigating a solution for this.

kayndo commented 2 years ago

I am facing the exact issue

roelvandenbrand commented 1 year ago

We are experiencing the same issue. The workaround not always works for us with doing the push locally.

austinamorusoyardstick commented 1 year ago

This is an issue I'm having as well. Is there a way to manually refresh the token?

austinamorusoyardstick commented 1 year ago

Nightwatch test are not finishing due timeout as well https://github.com/aws-amplify/amplify-hosting/issues/3204

ghost commented 1 year ago

Hi @austinamorusoyardstick 👋🏽 the 60 minute timeout is a hard limit at this time. Unfortunately there is no workaround for this but the team is aware of this limitation and are tracking this in our product backlog. We apologize for this inconvenience.

70ki8suda commented 1 year ago

facing same issue

localsecurity-emily commented 1 year ago

facing the same issue

gauravsapkal1 commented 1 year ago

@hloriii I am also facing same issue, I am using ISR and have around 25000 pages, my build takes more than hour, and i am getting same issue,

ExpiredToken: The provided token has expired. at Request.extractError (/root/.//node_modules/@sls-next/s3-static-assets/node_modules/aws-sdk/lib/services/s3.js:711:35) at Request.callListeners (/root/.//node_modules/@sls-next/s3-static-assets/node_modules/aws-sdk/lib/sequential_executor.js:106:20) at Request.emit (/root/.//node_modules/@sls-next/s3-static-assets/node_modules/aws-sdk/lib/sequential_executor.js:78:10)

@ymsrk your issue resolved ? if yes how ?

ymsrk commented 1 year ago

@gauravsapkal1 Let me tell you how we are getting around this problem. In my project, I limit the number of pages I create to around 25,000 so as not to exceed the 60 minute timeout. I would originally like to create more than 30,000 pages, but I accept the constraint. I hope this problem will be solved in the future.

gauravsapkal1 commented 1 year ago

@ymsrk Thanks for comment, I also creating around 25000 pages but now I will limit them upto 10,000.

gauravsapkal commented 1 year ago

@hloriii if we increase build time in environment variables "_BUILD_TIMEOUT" more than hour will it refresh token after an hour?

localsecurity-emily commented 1 year ago

@hloriii if we increase build time in environment variables "_BUILD_TIMEOUT" more than hour will it refresh token after an hour?

I can say from experience it will not refresh the token.

gauravsapkal commented 1 year ago

@localsecurity-emily yes I also thinking that it will not work, but I create case at aws they replied this @hloriii can you look into this

roelvandenbrand commented 1 year ago

That works up to one hour until the token expires. A timeout value will cut off the build if you want to make it last like less than 30 minutes max. When set to two hours, the token expiry is simply before that and fails on that. If that issues was not around, it would cut off build time at two hours and then stop.

gauravsapkal commented 1 year ago

@roelvandenbrand yes I tried its not working by increasing build time, @hloriii can you confirm that increase in _BUILD_TIMEOUT will refresh token or not.

Jay2113 commented 1 year ago

Hi everyone, thank you so much for your continued patience and apologies for the delayed response on this thread.

The ExpiredToken: The security token included in the request is expired exception can occur in the following scenarios:

Root cause:

Unfortunately, we do not have a way to increase the token expiration to more than an hour because this limitation is enforced by IAM on the role chaining duration: https://repost.aws/knowledge-center/iam-role-chaining-limit

During the backend build step:

When a backend build is initiated, Amplify Hosting’s build session role assumes your Amplify app’s IAM service role to deploy/update backend resources in your AWS account using temporary security credentials.

Possible workarounds:

It seems that long backend build duration (> 1hr) can typically occur if your app has a large schema defined in the amplify backend. Our recommendation would be to chunk out or split your changes in the schema. For example, if you were modifying/updating 4-5 relationships at once in your schema, try to update only a couple of relationships at once to see if that reduces the backend build duration.
Additionally, if your app requires some pre-build and/or post-build steps you can consider using a custom build environment as you can pre-install specific dependencies that usually take longer to install during a build using Amplify's default container and you can speed up this process by creating your own Docker image and reference it during a build.

During the frontend build step:

When a frontend build for a Web_Dynamic (Next.js 11) app (Classic SSR provider) is initiated, Amplify Hosting's build session role assumes your Amplify app's IAM service role to deploy/update backend resources in your AWS account using temporary security credentials.

Possible workarounds:

Our recommendation would be to reduce the total number of pages you are creating for your Next.js 11 app which thereby could reduce the build duration to < 1hr.
We strongly recommend that you migrate your Next.js 11 apps to the Amplify Hosting Compute managed SSR provider

Note: Updating the _BUILD_TIMEOUT Amplify environment variable to > 60 minutes will not refresh the security token.

djorgji commented 7 months ago

@Jay2113 We are having this is when trying to create new environment is created.

Is there any guidance on scenarios where there isn't an update happening but a create, and the create is timing out?

For us the main issue is the SearchStack taking close to 25 minutes to create, is there to tell it to skip that stack on the first build, or import an existing domain? We could maybe iterate and send a second build out manually, if we were able to somehow keep it from trying to deploy everything the first time around.

Even though that would suck, I would settle for any workaround as we try to figure out and move thing into CDK. We are completely stuck at the moment, our devs are completely blocked.

Thanks in advance.

pr0g commented 3 months ago

I believe our team has hit this issue as well. A developer on our team is attempting to create a new Amplify environment from our existing one, and the creation timed out. I suspect this is happening because we have a relatively complex GraphQL schema based on the advice above. Is there a potential workaround for this? @djorgji were you able to find a solution to this?

@Jay2113 if you have any suggestions I'd be very grateful to hear, thanks very much for your time!

aws-amplify / amplify-hosting