Deployment stuck at provision step

[steverandy]

Describe the bug Sometimes deployment stuck at the provision step and does not progress. There is no additional information displayed or an error message.

[swaminator]

Can you share your appid please? (Found in App settings > General)

[steverandy]

@swaminator d347fpgoiyxpzr, ap-southeast-2

[steverandy]

@swaminator build status is showing Pending and any subsequent webhook triggers appeared to be not processed.

[kahdojay]

@steverandy I see all the builds for d347fpgoiyxpzr in that region for 12/8 as being successful. Can you share the specific build number we should look at? Also if you have a way we can reproduce the issue, please let us know.

[steverandy]

Sure. Please check build no 367, 357, 355.

It usually happens for deployment triggered by webhook. Manually triggered deployment usually always successful.

[abouroubi]

Same bug here, Webhook started builds, always hangs on provisionning

[swaminator]

@abouroubi can you share your appid, build settings, and the build IDs that fail.

[abouroubi]

AppID : d3egu8kdaurpmi Build Settings :

version: 0.1
backend:
  phases:
    build:
      commands:
        - '# Execute Amplify CLI with the helper script'
        - amplifyPush --simple
frontend:
  phases:
    preBuild:
      commands:
        - npm ci
    build:
      commands:
        - npm run build
  artifacts:
    baseDirectory: public
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*

Build ID that hangs: Build #10

[kahdojay]

@abouroubi I see the build failed. If you visit the console are you able to see the build logs?

[kahdojay]

Sure. Please check build no 367, 357, 355.

It usually happens for deployment triggered by webhook. Manually triggered deployment usually always successful.

@steverandy I see these builds finished. If you go into the console after the fact, are you able to see them (i.e. now)? I wasn't able to reproduce - how long did you wait? If you can reproduce it again, does refreshing the console after some time show the right thing? Just trying to narrow it down.

[steverandy]

@kahdojay it does not show finished for me. It shows it only reaches provision step and I can't see any logs. I had to manually cancel this deployment and manually re-deploy in order to resume it. The screenshot above shows the state after I cancel the deployment manually.

Refreshing the console shows the same thing for me. I waited for more than 1 hour. We deploy multiple times a day and this issue is occurring every day for us.

[kahdojay]

@steverandy thanks. Just to confirm, this issue happens inconsistently for webhook-triggered deploys, whereas for manually triggered deploys, everything works fine?

Also, which branch? Release or master?

[steverandy]

@kahdojay yes that's correct.

The branch is release.

[abouroubi]

@kahdojay I had to cancel it and relaunch a new build

[kahdojay]

@abouroubi I see you had some successful builds afterward and were unable to unblock yourself.

To help us investigate further, were you able to reproduce the issue consistently, or is it the same as steverandy's situation where it happens inconsistently, and only for webhook-triggered builds?

[abouroubi]

@kahdojay just reproduced it yesterday, a Webook triggered the build, and it's stuck on provisionning.

I can unlock my self by cancelling and starting a new build but it's not sustanable, the webhooks are triggered when a new article is added to a Blog, I can't afford to give access to the amplify console to my copywriters, and tell them to check the builds each time they write a blog post

[kahdojay]

@abouroubi I understand it's not a sustainable solution, just trying to confirm whether it's the exact same issue as steverandy's since it will help us better understand the problem and come up with a fix. steverandy mentioned it happens inconsistently, i.e., sometimes the webhook-triggered build fails, but other times it works for a given branch. Is it the same experience for you?

[maxnatt]

Same issue with app at eu-central-1. Webhook-triggered build sometimes gets stuck at Provision stage, but always works after cancelling and starting it manually.

[abouroubi]

@kahdojay had to delete and create a new app, but the issue seems to appear on every webhook call for me.

[canofpopcom]

@kahdojay / @maxnatt / @abouroubi Same here just gets stuck on provisioning via the webhook and that then blocks updates from Git Branch, we had to revert to our fall back i.e. using: netlify.com which isn't ideal.

Do we think the issue will be resolved any time soon?

[steverandy]

@kahdojay do you have any updates on this issue?

[kahdojay]

@steverandy not at this time unfortunately, we are still investigating a fix.

[canofpopcom]

@kahdojay - HNY, we've got a site going live for CES 2020 tomorrow and we really need this to be resolved and an alternative workaround suggested (i.e. use an alternative datacentre) any ETA or are we to keep doing manual builds?

[kahdojay]

@canofpopcom no ETA yet, please bear with us and use the workaround for the time being while we continue investigating a fix

[canofpopcom]

@kahdojay - any news?

[canofpopcom]

@kahdojay is this now fixed?

[matteomelani]

Same issue here

[steverandy]

@kahdojay is there any update? thanks.

[husamuddin]

any update on this?

[gwryssekk]

@steverandy @husamuddin @canofpopcom @matteomelani @abouroubi @maxnatt

Hi All. Just wanted to update everyone that we have identified a fix for this issue and will be rolling it out soon.

[poolieweb]

Any update on the roll out of the release ?

[gwryssekk]

@steverandy @husamuddin @canofpopcom @matteomelani @abouroubi @maxnatt @poolieweb Hi all, this fix has been deployed for all regions. Feel free to test and let us know if you still see the issue.

[tkjaergaard]

Hi @gwryssekk,

I don't know if it's related, but I have just experienced that I'm getting stuck at the provisioning step when pushing to GitHub.

https://eu-central-1.console.aws.amazon.com/amplify/home?region=eu-central-1#/d3ab2d1gmwnk9b/dev/21

and

https://eu-central-1.console.aws.amazon.com/amplify/home?region=eu-central-1#/d3ab2d1gmwnk9b/master/14

[gwryssekk]

@tkjaergaard -- Hmm, ok, I'll check and see if it is related or if there is another similar issue.

[gwryssekk]

@tkjaergaard -- From the logs I have available, this doesn't seem to be the same issue. I see that later builds succeeded. After you encountered this, how did you proceed? Did you cancel the stuck job or just resubmit? Also, are you able to access the build logs from the console for these jobs?

[tkjaergaard]

@gwryssekk I click the cancel button form within the UI and the pressed the "redeploy this version"-button immediately after. Nothing but the provisioning-logs was available. Like it never got further.

[gwryssekk]

Thanks, @tkjaergaard....I'll dig a bit more and see what I can find. After you cancelled and redeployed did it succeed?

[tkjaergaard]

@gwryssekk Yes, it did. It went right through without any hiccups.

[madsrasmussen]

Hey @gwryssekk,

I am experiencing the same issue with provisioning. The URL to Amplify: https://eu-central-1.console.aws.amazon.com/amplify/home?region=eu-central-1#/d3o9mdzy9l1hu2/master/100 . I canceled the deployment and clicked the "Redeploy this version"-button. After that everything went smoothly.

[gwryssekk]

Hi folks, thanks for the updates here. = While we did fix one issue that was causing this to occur it seems there is another one we have to track down. I'm going to create a task to have the team look into this in detail.

[tkjaergaard]

@gwryssekk Just experienced the issue again today on another Amplify project.

https://eu-central-1.console.aws.amazon.com/amplify/home?region=eu-central-1#/d2dldaj4mfy118/dev/116?step=PROVISION

[behrooziAWS]

@tkjaergaard we had an issue with screenshotting that was impacting builds during that window (https://github.com/aws-amplify/amplify-console/issues/948). We've resolved the underlying issue so builds shouldn't be blocked. It looks like subsequent builds have succeeded for you. Was this a one-off that occurred only in the last week or so or is this a recurring issue for you?

[harishletsgo]

Provisioning takes a very long time. Have the same issue. Took at least 18 Mins.

[judygab]

@kahdojay it does not show finished for me. It shows it only reaches provision step and I can't see any logs. I had to manually cancel this deployment and manually re-deploy in order to resume it. The screenshot above shows the state after I cancel the deployment manually.

Refreshing the console shows the same thing for me. I waited for more than 1 hour. We deploy multiple times a day and this issue is occurring every day for us.

I'm having the same issue, provisioning step is taking too long, how were you able to cancel it?

[maksym-bordun-ynd]

Are there any news? Fix was deployed about 5 months ago, but builds still get stuck on Provision from time to time (and always work when build gets restarted manually), as described in the first message. At least when talking about eu-central-1.

[shide1989]

@gwryssekk This issue is still happing while the fix has been deployed months ago. Any update ?

[gregeatworld]

Same here. Really very frustrating situation as for paid service. Can we count on soonest fix or at least any details regarding this?

[stevechanvii]

Same here. Really very frustrating situation as for paid service. Can we count on soonest fix or at least any details regarding this?

Have same problem this couple of weeks (region Sydney). Some times stuck at provision, once I click cancel the status of provision change from pending to success, but other steps failed. Manually rebuild the project would successfully build the project.

[litwicki]

@stevechanvii @gregeatworld @shide1989 @maksym-bordun-ynd @judygab We are revisiting this issue this week and will update here as soon as we have more information.

[talhaabbasi]

Same issue for me:

@litwicki any update on this?