Amplify root stack stuck at UPDATE_ROLLBACK_FAILED state

[parvusville]

Before opening, please confirm:

• [X] I have installed the latest version of the Amplify CLI (see above), and confirmed that the issue still persists.
• [X] I have searched for duplicate or closed issues.
• [X] I have read the guide for submitting bug reports.
• [X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
• [X] I have removed any sensitive information from my code snippets and submission.

How did you install the Amplify CLI?

npm i -g @aws-amplify/cli

If applicable, what version of Node.js are you using?

v14.17.6

Amplify CLI Version

7.6.7

What operating system are you using?

Pop Os

Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.

No manual changes

Amplify Categories

function, api

Amplify Commands

push

Describe the bug

My environment ended up in UPDATE_ROLLBACK_FAILED state, which completely blocks me from doing any updates.

Initially I did a casual amplify push (at this point using amplify cli 7.6.7) and faced https://github.com/aws-amplify/amplify-category-api/issues/715, which caused UPDATE_FAILED Embedded stack arn:aws:cloudformation:eu-west-1:12345:stack/amplify-myapp-devtest-12345-apimyapp-ABCD/abcd-1234 was not successfully updated. Currently in UPDATE_ROLLBACK_IN_PROGRESS with reason: The following resource(s) failed to update: [SearchableStack]. Which then caused update rollback to fail, because the python version used on searchable lambda is no longer supported I believe. UPDATE_ROLLBACK_FAILED The following resource(s) failed to update: [apimyapp].

At the nested SearchableStack I see: ElasticSearchStreamingLambdaFunction UPDATE_FAILED Resource handler returned message: "The runtime parameter of python3.6 is no longer supported for creating or updating AWS Lambda functions. We recommend you use the new runtime (python3.9) while creating or updating functions. (Service: Lambda, Status Code: 400, Request ID: 0f4011a3-1c87-4b1c-ae10-f9513cef928c)" (RequestToken: 9cf22463-9951-27b6-277d-f9cb7308f58c, HandlerErrorCode: InvalidRequest) amplify-myapp-devtest-160256-apimyapp-ABCD-SearchableStack-1DT6WJCWFDDIJ UPDATE_ROLLBACK_FAILED The following resource(s) failed to update: [ElasticSearchStreamingLambdaFunction].

My problem is that I haven't been able to recover from this UPDATE_ROLLBACK_FAILED state. If I try to skip the affected resource (SearchableStack) as in the following link, the rollback again fails: UPDATE_ROLLBACK_FAILED Nested stack resources can only be skipped when their embedded stack statuses are one of [DELETE_COMPLETE, DELETE_IN_PROGRESS, DELETE_FAILED] https://aws.amazon.com/premiumsupport/knowledge-center/cloudformation-update-rollback-failed/

Searching the discord I pretty much only found "I had to delete the environment" on similar occasions, which surely cannot be an answer here? :sweat:

Expected behavior

Amplify app should be recoverable from UPDATE_ROLLBACK_FAILED state, and preferably not end up in one.

Reproduction steps

1. amplify push with v1 transformer that uses python 3.6 for DdbToEsFn

This seems to be enought to get the whole environment stuck.

Additional information

I haven't yet figured out a way to get out of UPDATE_ROLLBACK_FAILED state in order to update the stack with Python 3.9

[AaronZyLee]

You can try to add the following line in the parameters.json under amplify/backend/api/${api_name} and re-run the amplify push to fix the runtime deprecated issue.

{
  "ElasticSearchStreamingLambdaRuntime": "python3.9"
}

[parvusville]

@AaronZyLee I saw that as a recommended solution, but I am unable to push. I am completely blocked because of the UPDATE_ROLLBACK_FAILED -state.

[taoatmars]

I am also facing the same problem at the production level. This happens unexpectedly, I think AWS took out the ability to create/update lambda in 3.6, therefore, the lambda can't even rollback.

[simoneagostinelli]

@parvusville i solved the UPDATE_ROLLBACK_FAILED directly on cloudformation, first you have to go on your lambda stream function the name sould be something like this 'DdbToEsFn-.... and in the code tab change the python runtime to 3.9, after that, go back to the cloudformation root stack and click on stack actions and choose continue update rollback, in the dialog leave unchecked the api resource, now the rollback should work.

[parvusville]

Thanks @simoneagostinelli , this worked.

[josefaidt]

Closing as a duplicate of https://github.com/aws-amplify/amplify-category-api/issues/715