I made changes to my dev environment, merged to main, and got UPDATE_COMPLETE_CLEANUP_IN_PROGRESS state and can not be updated

[Bryson14]

How did you install the Amplify CLI?

npm

If applicable, what version of Node.js are you using?

v20.10.0

Amplify CLI Version

12.10.1

What operating system are you using?

ubuntu

Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.

no manual changes

Describe the bug

I made the following changes in my dev environment:

1. added 4 GSI's
2. Made some ui and structural changes
3. Deleted a function written in go and remade it with node

Everything deployed (with amplify push) and published (with continuous cicd) fine on dev. I then merged to main and tried to run amplify push but got the following errors:

⠸ Building resource api/myapp✅ GraphQL schema compiled successfully.

Edit your schema at /home/me/Code/consulting_projects/tmt/myapp/amplify/backend/api/myapp/schema.graphql or place .graphql files in a directory at /home/me/Code/consulting_projects/tmt/myapp/amplify/backend/api/myapp/schema
✔ Successfully pulled backend environment main from the cloud.
⠙ Building resource api/myapp✅ GraphQL schema compiled successfully.

Edit your schema at /home/brymei/Code/consulting_projects/tmt/myapp/amplify/backend/api/myapp/schema.graphql or place .graphql files in a directory at /home/me/Code/consulting_projects/tmt/myapp/amplify/backend/api/myapp/schema

    Current Environment: main

┌──────────┬────────────────────────────────┬───────────┬───────────────────┐
│ Category │ Resource name                  │ Operation │ Provider plugin   │
├──────────┼────────────────────────────────┼───────────┼───────────────────┤
│ Hosting  │ amplifyhosting                 │ Create    │                   │
├──────────┼────────────────────────────────┼───────────┼───────────────────┤
│ Api      │ notificationapi                │ Update    │ awscloudformation │
├──────────┼────────────────────────────────┼───────────┼───────────────────┤
│ Api      │ tmtamplifyapp                  │ Update    │ awscloudformation │
├──────────┼────────────────────────────────┼───────────┼───────────────────┤
│ Function │ notificationScheduledSender    │ Update    │ awscloudformation │
├──────────┼────────────────────────────────┼───────────┼───────────────────┤
│ Function │ scheduledDataCleaner           │ Update    │ awscloudformation │
├──────────┼────────────────────────────────┼───────────┼───────────────────┤
│ Function │ sendNotificationGo             │ Update    │ awscloudformation │
├──────────┼────────────────────────────────┼───────────┼───────────────────┤
│ Api      │ AdminQueries                   │ No Change │ awscloudformation │
├──────────┼────────────────────────────────┼───────────┼───────────────────┤
│ Auth     │ tmtAmplifyApp                  │ No Change │ awscloudformation │
├──────────┼────────────────────────────────┼───────────┼───────────────────┤
│ Function │ AdminQueries38301bfb           │ No Change │ awscloudformation │
├──────────┼────────────────────────────────┼───────────┼───────────────────┤
│ Storage  │ s3tmtamplifyappstoragec3cc73b4 │ No Change │ awscloudformation │
└──────────┴────────────────────────────────┴───────────┴───────────────────┘
✔ Are you sure you want to continue? (Y/n) · yes
✅ GraphQL schema compiled successfully.

Edit your schema at /home/me/Code/consulting_projects/tmt/myapp/amplify/backend/api/myapp/schema.graphql or place .graphql files in a directory at /home/me/Code/consulting_projects/tmt/myapp/amplify/backend/api/myapp/schema
⠏ Building resource api/tmtamplifyapp✅ GraphQL schema compiled successfully.

Edit your schema at /home/me/Code/consulting_projects/tmt/myapp/amplify/backend/api/myapp/schema.graphql or place .graphql files in a directory at /home/me/Code/consulting_projects/tmt/myapp/amplify/backend/api/myapp/schema
⚠️ Warning: You've specified "*" as the "Resource" in sendNotificationGo's custom IAM policy.
 This will grant sendNotificationGo the ability to perform ses:SendEmail,ses:SendRawEmail on ALL resources in this AWS Account.
⚠️ Warning: You've specified "*" as the "Resource" in sendNotificationGo's custom IAM policy.
 This will grant sendNotificationGo the ability to perform SNS:Publish on ALL resources in this AWS Account.
Deployment completed.

Rolled back (2 of 1)
🛑 ["Index: 0 State: {\"deploy\":\"deploying\"} Message: Stack:arn:aws:cloudformation:us-east-1:XXXXXXX:stack/amplify-tmtamplifyapp-main-183644-apitmtamplifyapp-1OB0VC3EYWXXH/44e74640-c1f1-11ed-b081-0aaebdb172df is in UPDATE_COMPLETE_CLEANUP_IN_PROGRESS state and can not be updated."]

Learn more at: https://docs.amplify.aws/cli/project/troubleshooting/

Session Identifier: ab7b4a94-753c-4623-b6b0-de4d34343860

✅ Report saved: /tmp/tmtAmplifyApp/report-1707071616199.zip

✔ Done

Expected behavior

i expected the push to take a while since it was making GSI's but then it failed.

I ran amplify push --iterative-rollback to try to revert this error but it resulted in the same console output.

Here is the output of the stacks

This is the detected root cause for the root stack for the app:

Reproduction steps

Unsure how to replicate.

Project Identifier

be829bf2e298a56012f84ebe8815edf2

Log output

``` # Put your logs below this line ```

Additional information

No response

Before submitting, please confirm:

• [X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
• [X] I have removed any sensitive information from my code snippets and submission.

[ykethan]

Hey,👋 thanks for raising this! I'm going to transfer this over to our API repository for better assistance 🙂

[Bryson14]

On the Amplify Studio, it should UPDATE_FAILED for my GraphQL api

API: cloudFormation User: arn:aws:sts::XXX:assumed-role/us-east-1_WjM1TuhLB_Full-access/amplifyadmin is not authorized to perform: cloudformation:RollbackStack on resource: arn:aws:cloudformation:us-east-1:XXX:stack/amplify-tmtamplifyapp-main-183644-apitmtamplifyapp-1OB0VC3EYWXXH/44e74640-c1f1-11ed-b081-0aaebdb172df because no identity-based policy allows the cloudformation:RollbackStack action

and then for the main stack, It saying ROLLBACK_FAILED

The following resource(s) failed to update: [apitmtamplifyapp, apinotificationapi].

This is getting ridiculous. i've been trying to fix this, but this is my production environment.

[Bryson14]

I attempted to manually refresh the stack because I thought that the IAM role with amplify doesn't have enough permisions. That did allow me to run amplify push again, but then push failed after about 30 minutes.

Looking in the cloud formation templates, the core issue is that the template is trying to setup the /email route on the API gateway, but that doesn't exist anymore in the code. i've ran amplify remove function and amplify remove api to get rid of both the function and the api route from the code, but I cannot run amplify push successfully to remove this faulty stack. What can I do?