Amplify CLI push fails to push to environment after removing env vars from function

u93 commented 1 year ago

How did you install the Amplify CLI?

npm

If applicable, what version of Node.js are you using?

14.21.3

Amplify CLI Version

10.7.3

What operating system are you using?

Mac

Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.

No

Describe the bug

Updated function by removing environment variables and also pushed to my dev environment with amplify push. After moving to a different environment by changing branches and running amplify env checkout {ENV_NAME} and doing, amplify push again, the deployment fails

        functionapi                    AWS::CloudFormation::Stack     UPDATE_FAILED                  Mon Feb 27 2023 09:14:3
Deployed function AdminQueriesd8df3ab6 [ ======================================== ] 3/3
Deployed function S3Triggerc2f3c2fa [ ======================================== ] 3/3
Deployed function api [ ======================================== ] 5/5
Deployed function coreconsolecf3573d0PostAuthentication [ ======================================== ] 6/6

🛑 The following resources failed to deploy:
Resource Name: functionapi (AWS::CloudFormation::Stack)
Event Type: update
Reason: Parameters: [telemetryCoreCurrentStateTableBaseName, telemetryCamerasTableBaseName, videoCameraCoreTableBaseName, permissionsTableName, eventsTableBaseName, ddnsCameraTableBaseName, placeholder, notesTableBaseName, sessionsTableName, transcodedCameraCoreTableBaseName, aiCameraCoreTableBaseName, telemetryCoreTableBaseName] do not exist in the template
URL: https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/arn%3Aaws%3Acloudformation%3Aus-east-1%3A217822356980%3Astack%2Famplify-coreconsole-release-193550%2Fd06dfc00-256b-11ec-8516-0e6c55b6e46f/events

🛑 Resource is not in the state stackUpdateComplete
Name: functionapi (AWS::CloudFormation::Stack), Event Type: update, Reason: Parameters: [telemetryCoreCurrentStateTableBaseName, telemetryCamerasTableBaseName, videoCameraCoreTableBaseName, permissionsTableName, eventsTableBaseName, ddnsCameraTableBaseName, placeholder, notesTableBaseName, sessionsTableName, transcodedCameraCoreTableBaseName, aiCameraCoreTableBaseName, telemetryCoreTableBaseName] do not exist in the template

The error produced a session ID and also a report. Happy to share them just need to know how.

Expected behavior

After checkout of the env that I want to deploy, deployment should be successful as it has been always when doing something similar.

Reproduction steps

Update function with amplify function update
Select remove env vars and remove some of them
Push with amplify push
Move to a different branch and do amplify env checkout release
Push release environment amplify push

Project Identifier

4d3029e4c61cca9982ff470ba7f8b44e

Log output

``` # Put your logs below this line (base) eebf1993@MacBook-Pro-4 coreconsole % amplify push ✔ Successfully pulled backend environment release from the cloud. Current Environment: release ┌───────────┬───────────────────────────────────────┬───────────┬───────────────────┐ │ Category │ Resource name │ Operation │ Provider plugin │ ├───────────┼───────────────────────────────────────┼───────────┼───────────────────┤ │ Function │ api │ Update │ awscloudformation │ ├───────────┼───────────────────────────────────────┼───────────┼───────────────────┤ │ Analytics │ coreconsole │ No Change │ awscloudformation │ ├───────────┼───────────────────────────────────────┼───────────┼───────────────────┤ │ Api │ AdminQueries │ No Change │ awscloudformation │ ├───────────┼───────────────────────────────────────┼───────────┼───────────────────┤ │ Api │ supportui │ No Change │ awscloudformation │ ├───────────┼───────────────────────────────────────┼───────────┼───────────────────┤ │ Auth │ coreconsolecf3573d0 │ No Change │ awscloudformation │ ├───────────┼───────────────────────────────────────┼───────────┼───────────────────┤ │ Auth │ userPoolGroups │ No Change │ awscloudformation │ ├───────────┼───────────────────────────────────────┼───────────┼───────────────────┤ │ Function │ AdminQueriesd8df3ab6 │ No Change │ awscloudformation │ ├───────────┼───────────────────────────────────────┼───────────┼───────────────────┤ │ Function │ S3Triggerc2f3c2fa │ No Change │ awscloudformation │ ├───────────┼───────────────────────────────────────┼───────────┼───────────────────┤ │ Function │ coreconsolecf3573d0PostAuthentication │ No Change │ awscloudformation │ ├───────────┼───────────────────────────────────────┼───────────┼───────────────────┤ │ Hosting │ S3AndCloudFront │ No Change │ awscloudformation │ ├───────────┼───────────────────────────────────────┼───────────┼───────────────────┤ │ Storage │ fususcoreConsoleEvents │ No Change │ awscloudformation │ ├───────────┼───────────────────────────────────────┼───────────┼───────────────────┤ │ Storage │ fususcoreConsoleSettings │ No Change │ awscloudformation │ ├───────────┼───────────────────────────────────────┼───────────┼───────────────────┤ │ Storage │ supportUiContent │ No Change │ awscloudformation │ └───────────┴───────────────────────────────────────┴───────────┴───────────────────┘ ? Are you sure you want to continue? Yes ⚠️ Warning: You've specified "*" as the "Resource" in api's custom IAM policy. This will grant api the ability to perform secretsmanager:GetRandomPassword,secretsmanager:ListSecrets on ALL resources in this AWS Account. Deployment failed. Deploying root stack coreconsole [ ==================================------ ] 12/14 amplify-coreconsole-release-1… AWS::CloudFormation::Stack UPDATE_ROLLBACK_COMPLETE Mon Feb 27 2023 09:15:0 storagefususcoreConsoleSettin… AWS::CloudFormation::Stack UPDATE_COMPLETE Mon Feb 27 2023 09:14:4 storagefususcoreConsoleEvents AWS::CloudFormation::Stack UPDATE_COMPLETE Mon Feb 27 2023 09:14:4 analyticscoreconsole AWS::CloudFormation::Stack UPDATE_COMPLETE Mon Feb 27 2023 09:14:4 hostingS3AndCloudFront AWS::CloudFormation::Stack UPDATE_COMPLETE Mon Feb 27 2023 09:14:4 functionS3Triggerc2f3c2fa AWS::CloudFormation::Stack UPDATE_COMPLETE Mon Feb 27 2023 09:14:4 functioncoreconsolecf3573d0Po… AWS::CloudFormation::Stack UPDATE_COMPLETE Mon Feb 27 2023 09:14:4 authcoreconsolecf3573d0 AWS::CloudFormation::Stack UPDATE_COMPLETE Mon Feb 27 2023 09:14:4 authuserPoolGroups AWS::CloudFormation::Stack UPDATE_COMPLETE Mon Feb 27 2023 09:14:5 functionAdminQueriesd8df3ab6 AWS::CloudFormation::Stack UPDATE_COMPLETE Mon Feb 27 2023 09:14:5 apiAdminQueries AWS::CloudFormation::Stack UPDATE_COMPLETE Mon Feb 27 2023 09:14:5 storagesupportUiContent AWS::CloudFormation::Stack UPDATE_COMPLETE Mon Feb 27 2023 09:14:5 functionapi AWS::CloudFormation::Stack UPDATE_FAILED Mon Feb 27 2023 09:14:3 Deployed function AdminQueriesd8df3ab6 [ ======================================== ] 3/3 Deployed function S3Triggerc2f3c2fa [ ======================================== ] 3/3 Deployed function api [ ======================================== ] 5/5 Deployed function coreconsolecf3573d0PostAuthentication [ ======================================== ] 6/6 🛑 The following resources failed to deploy: Resource Name: functionapi (AWS::CloudFormation::Stack) Event Type: update Reason: Parameters: [telemetryCoreCurrentStateTableBaseName, telemetryCamerasTableBaseName, videoCameraCoreTableBaseName, permissionsTableName, eventsTableBaseName, ddnsCameraTableBaseName, placeholder, notesTableBaseName, sessionsTableName, transcodedCameraCoreTableBaseName, aiCameraCoreTableBaseName, telemetryCoreTableBaseName] do not exist in the template URL: https://console.aws.amazon.com/cloudformation/home?region=us-east-1#/stacks/arn%3Aaws%3Acloudformation%3Aus-east-1%3A217822356980%3Astack%2Famplify-coreconsole-release-193550%2Fd06dfc00-256b-11ec-8516-0e6c55b6e46f/events 🛑 Resource is not in the state stackUpdateComplete Name: functionapi (AWS::CloudFormation::Stack), Event Type: update, Reason: Parameters: [telemetryCoreCurrentStateTableBaseName, telemetryCamerasTableBaseName, videoCameraCoreTableBaseName, permissionsTableName, eventsTableBaseName, ddnsCameraTableBaseName, placeholder, notesTableBaseName, sessionsTableName, transcodedCameraCoreTableBaseName, aiCameraCoreTableBaseName, telemetryCoreTableBaseName] do not exist in the template ```

Additional information

No response

Before submitting, please confirm:

[X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
[X] I have removed any sensitive information from my code snippets and submission.

u93 commented 1 year ago

As a note, this issue it's stopping a production release from our application and somehow I'm still getting the AMPLIFY_parameter resources again after I stopped getting them

 Category   │ Resource name                                                               │ Operation │ Provider plugin   │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Parameters │ AMPLIFY_function_AdminQueriesd8df3ab6_deploymentBucketName                  │ Create    │                   │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Parameters │ AMPLIFY_function_AdminQueriesd8df3ab6_s3Key                                 │ Create    │                   │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Parameters │ AMPLIFY_function_S3Triggerc2f3c2fa_deploymentBucketName                     │ Create    │                   │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Parameters │ AMPLIFY_function_S3Triggerc2f3c2fa_s3Key                                    │ Create    │                   │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Parameters │ AMPLIFY_function_api_aiCameraCoreTableBaseName                              │ Create    │                   │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Parameters │ AMPLIFY_function_api_cognitoUserPoolClientId                                │ Create    │                   │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Parameters │ AMPLIFY_function_api_cognitoUserPoolId                                      │ Create    │                   │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Parameters │ AMPLIFY_function_api_ddnsCameraTableBaseName                                │ Create    │                   │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Parameters │ AMPLIFY_function_api_deploymentBucketName                                   │ Create    │                   │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Parameters │ AMPLIFY_function_api_env                                                    │ Create    │                   │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Parameters │ AMPLIFY_function_api_eventsTableBaseName                                    │ Create    │                   │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Parameters │ AMPLIFY_function_api_notesTableBaseName                                     │ Create    │                   │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Parameters │ AMPLIFY_function_api_permissionsTableName                                   │ Create    │                   │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Parameters │ AMPLIFY_function_api_placeholder                                            │ Create    │                   │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Parameters │ AMPLIFY_function_api_region                                                 │ Create    │                   │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Parameters │ AMPLIFY_function_api_s3Key                                                  │ Create    │                   │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Parameters │ AMPLIFY_function_api_sessionsTableName                                      │ Create    │                   │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Parameters │ AMPLIFY_function_api_telemetryCamerasTableBaseName                          │ Create    │                   │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Parameters │ AMPLIFY_function_api_telemetryCoreCurrentStateTableBaseName                 │ Create    │                   │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Parameters │ AMPLIFY_function_api_telemetryCoreTableBaseName                             │ Create    │                   │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Parameters │ AMPLIFY_function_api_transcodedCameraCoreTableBaseName                      │ Create    │                   │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Parameters │ AMPLIFY_function_api_userAccessKey                                          │ Create    │                   │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Parameters │ AMPLIFY_function_api_userRegion                                             │ Create    │                   │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Parameters │ AMPLIFY_function_api_userSecretKey                                          │ Create    │                   │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Parameters │ AMPLIFY_function_api_videoCameraCoreTableBaseName                           │ Create    │                   │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Parameters │ AMPLIFY_function_coreconsolecf3573d0PostAuthentication_deploymentBucketName │ Create    │                   │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Parameters │ AMPLIFY_function_coreconsolecf3573d0PostAuthentication_s3Key                │ Create    │                   │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Analytics  │ coreconsole                                                                 │ No Change │ awscloudformation │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Api        │ AdminQueries                                                                │ No Change │ awscloudformation │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Api        │ supportui                                                                   │ No Change │ awscloudformation │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Auth       │ coreconsolecf3573d0                                                         │ No Change │ awscloudformation │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Auth       │ userPoolGroups                                                              │ No Change │ awscloudformation │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Function   │ AdminQueriesd8df3ab6                                                        │ No Change │ awscloudformation │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Function   │ S3Triggerc2f3c2fa                                                           │ No Change │ awscloudformation │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Function   │ api                                                                         │ No Change │ awscloudformation │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Function   │ coreconsolecf3573d0PostAuthentication                                       │ No Change │ awscloudformation │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Hosting    │ S3AndCloudFront                                                             │ No Change │ awscloudformation │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Storage    │ fususcoreConsoleEvents                                                      │ No Change │ awscloudformation │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Storage    │ fususcoreConsoleSettings                                                    │ No Change │ awscloudformation │
├────────────┼─────────────────────────────────────────────────────────────────────────────┼───────────┼───────────────────┤
│ Storage    │ supportUiContent                                                            │ No Change │ awscloudformation │
└────────────┴─────────────────────────────────────────────────────────────────────────────┴───────────┴───────────────────┘
[➥] Resource Stack: Parameters/AMPLIFY_function_AdminQueriesd8df3ab6_deploymentBucketName : Create
No changes  
[➥] Resource Stack: Parameters/AMPLIFY_function_AdminQueriesd8df3ab6_s3Key : Create
No changes  
[➥] Resource Stack: Parameters/AMPLIFY_function_S3Triggerc2f3c2fa_deploymentBucketName : Create
No changes  
[➥] Resource Stack: Parameters/AMPLIFY_function_S3Triggerc2f3c2fa_s3Key : Create
No changes  
[➥] Resource Stack: Parameters/AMPLIFY_function_api_aiCameraCoreTableBaseName : Create
No changes  
[➥] Resource Stack: Parameters/AMPLIFY_function_api_cognitoUserPoolClientId : Create
No changes  
[➥] Resource Stack: Parameters/AMPLIFY_function_api_cognitoUserPoolId : Create
No changes  
[➥] Resource Stack: Parameters/AMPLIFY_function_api_ddnsCameraTableBaseName : Create
No changes  
[➥] Resource Stack: Parameters/AMPLIFY_function_api_deploymentBucketName : Create
No changes  
[➥] Resource Stack: Parameters/AMPLIFY_function_api_env : Create
No changes  
[➥] Resource Stack: Parameters/AMPLIFY_function_api_eventsTableBaseName : Create
No changes  
[➥] Resource Stack: Parameters/AMPLIFY_function_api_notesTableBaseName : Create
No changes  
[➥] Resource Stack: Parameters/AMPLIFY_function_api_permissionsTableName : Create
No changes  
[➥] Resource Stack: Parameters/AMPLIFY_function_api_placeholder : Create
No changes  
[➥] Resource Stack: Parameters/AMPLIFY_function_api_region : Create
No changes  
[➥] Resource Stack: Parameters/AMPLIFY_function_api_s3Key : Create
No changes  
[➥] Resource Stack: Parameters/AMPLIFY_function_api_sessionsTableName : Create
No changes  
[➥] Resource Stack: Parameters/AMPLIFY_function_api_telemetryCamerasTableBaseName : Create
No changes  
[➥] Resource Stack: Parameters/AMPLIFY_function_api_telemetryCoreCurrentStateTableBaseName : Create
No changes  
[➥] Resource Stack: Parameters/AMPLIFY_function_api_telemetryCoreTableBaseName : Create
No changes  
[➥] Resource Stack: Parameters/AMPLIFY_function_api_transcodedCameraCoreTableBaseName : Create
No changes  
[➥] Resource Stack: Parameters/AMPLIFY_function_api_userAccessKey : Create
No changes  
[➥] Resource Stack: Parameters/AMPLIFY_function_api_userRegion : Create
No changes  
[➥] Resource Stack: Parameters/AMPLIFY_function_api_userSecretKey : Create
No changes  
[➥] Resource Stack: Parameters/AMPLIFY_function_api_videoCameraCoreTableBaseName : Create
No changes  
[➥] Resource Stack: Parameters/AMPLIFY_function_coreconsolecf3573d0PostAuthentication_deploymentBucketName : Create
No changes  
[➥] Resource Stack: Parameters/AMPLIFY_function_coreconsolecf3573d0PostAuthentication_s3Key : Create
No changes

And in the environment that is failing, those parameters do not show up, which I assume it's what is causing the issue... Can I manually delete them? Or is there a way to get those to stop showing?

Thanks!

u93 commented 1 year ago

If it helps, here is one of the session IDs that amplify CLI has shown for these errors:

518b483d-c3ba-4f07-9e1e-e1cdc544cbc3

Thanks!

josefaidt commented 1 year ago

Hey @u93 :wave: thanks for raising this! When changing Amplify environments are you also checking out a different git branch? Can you try running amplify env checkout release twice before pushing?

The Parameters you see were introduced in a later version of the CLI and will be removed from your status print after you're able to successfully push with the CLI.

Is functionapi the Lambda function you updated? Are the missing parameters related to the resource access permissions it has such as Table access?

u93 commented 1 year ago

Hi @josefaidt,

My process for pushing to different environments it's the same as the one described in the amplify docs... First will do my changes to my dev env from my dev branch, then push them. After will do a merge to the new branch, release branch in this case, then a checkout of the release branch and then for last amplify env checkout release to then push...

The changes related are for environment variables configurations, there were removed from the dev version function and that somehow are required in release still. The parameters are also shown in team-provider-info.json

Doing amplify env checkout release twice does not do anything...

Regards

u93 commented 1 year ago

Also, @josefaidt , in which files should those parameters be located... In backend-config.json only or in none at all? The reason I ask, is that when I'm going to make a push to my dev environment, which is an environment that is working right now, I get changes in my repo, in backend-config.json adding the AMPLIFY_function_xxxx parameters, then when I merge into my release branch and try to push, I get the opposite, Amplify CLI tries to remove the AMPLIFY_function_xxxx parameters

josefaidt commented 1 year ago

Hey @u93 environment-specific parameters are now stored in SSM Parameter Store and in the team-provider-info.json file. These parameters (shown as "Parameter" resources) should be stored in amplify/backend/backend-config.json. If they are also present in amplify-meta.json it should be safe to remove them rather than waiting for the successful push, however their display in the status page is a cosmetic bug and should not effect the success of amplify push.

I will take a deeper look at these parameters across environments

u93 commented 1 year ago

Hi @josefaidt ,

Thanks for digging and the explanation. Do you need more from information from me? As I said before, this issue it's impairing our production deployment of the application and we do not know how to pass this as it's not an issue in our code and the CLI was used in a very standard way.

Is there any way to override those parameters so the cloudformation stack does not need them?

Regards,

josefaidt commented 1 year ago

Hey @u93 I was able to reproduce this by using the provided steps. Marking as a bug.

As a workaround, you can run manually remove the environment variables from the corresponding function block in your amplify/team-provider-info.json file and remove the associated parameters from amplify/backend/backend-config.json

parvusville commented 1 year ago

I am facing the same behavior as @u93, among some other problems at #12188. When I am merging my changes from dev environment to staging environment and trying to do amplify push, CLI for some reason removes the whole parameters block from my backend-config.json. Is it safe to complete the push, or is there something I can do to workaround this @josefaidt? I guess it is not okay that in staging Amplify removes this part as it remains there in dev.

I don't want to risk breaking the environment, so this is also blocking releases for us now.

E: As a sidenote, I have not experienced the visual bug where all parameters are shown on amplify status. E2: Apparently the parameters block being removed on push is by design. It is added back after completing the push.

aws-amplify / amplify-cli