Old secret value deleted and new lambda dissapears when pushing changes (with new secrets) into another environment.

parvusville commented 1 year ago

How did you install the Amplify CLI?

npm

If applicable, what version of Node.js are you using?

v14.17.6

Amplify CLI Version

10.8.1

What operating system are you using?

Pop Os

Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.

No

Describe the bug

Merging and pushing changes from dev to staging caused an old secret value to be deleted, and newly added function not being created.

Things done in dev environment:

Added a new Lambda function with an environment variable and a secret
Added one new environment variable to existing old Lambda function
Added two new secrets to existing old Lambda function
Other non-cli things such as additions to graphql schema and modifications to Function code

Everything works great. Then moving changes into staging: Not 100% sure if 1 and 2 were in this order

amplify env checkout staging
git checkout staging
git merge dev
amplify push
Add missing env variables and secrets
From git diff I see that I now have a missing secret on the old lambda, and the new lambda does not exist at all:

In the Old lambda one secret that existed before is now removed from oldLambda-cloudformation-template.json , function-parameters.json and also the automatically generated example of getting the secrets in the Lambdas index.js. From parameter store I can see that it is actually removed.

For the New lambda, in backend-config.json I see that following parts are removed:

AMPLIFY_function_newLambda_deploymentBucketName
AMPLIFY_function_newLambda_s3Key
AMPLIFY_function_newLambda_secretsPathAmplifyAppId

Doing amplify status on staging does not list the new function at all. I see that there are still some references left:

  "function": {
    "newLambda": {
      "build": true,
      "dependsOn": [
        {
          "attributes": [
            "GraphQLAPIIdOutput"
          ],
          "category": "api",
          "resourceName": "urentdealer"
        }
      ],
      "providerPlugin": "awscloudformation",
      "service": "Lambda"
    },

....

    "AMPLIFY_function_newLambda_myEnvVar": {
      "usedBy": [
        {
          "category": "function",
          "resourceName": "newLambda"
        }
      ]
    },

Expected behavior

Existing secrets should not disappear and newly added Lamba should be created, when pushing changes into another enviornment

Reproduction steps

Add multiple secret values for existing Lambda function, which already has one secret
Add a new Lambda function
Release changes in dev
amplify env checkout staging
git checkout staging
git merge dev
amplify push
Add missing env variables and secrets
From git diff I see that I now have a missing secret on the old lambda, and the new lambda does not exist at all

Project Identifier

6b0d8978f235dda0a7c441c51bcf218f

Log output

``` # Put your logs below this line ```

Additional information

No response

Before submitting, please confirm:

[X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
[X] I have removed any sensitive information from my code snippets and submission.

parvusville commented 1 year ago

I can re-add the disappeared secret value by reverting everything done after the merge to staging, and then doing amplify function update -> update secret values -> selecting the deleted secret. This re-creates it.

However when trying to do the merge again, I always seem to run into the same problems. Doing the merge again I noticed that amplify status results into the changes seen in backend-config.json. This time only reference left to new lambda is this:

    "newLambda": {
      "build": true,
      "dependsOn": [
        {
          "attributes": [
            "GraphQLAPIIdOutput"
          ],
          "category": "api",
          "resourceName": "urentdealer"
        }
      ],
      "providerPlugin": "awscloudformation",
      "service": "Lambda"
    },

Now also the environment variable I in dev added to the Old lambda is removed from backend-config.json, and I don't see it when doing amplify function update ➝ update env variable in staging.

If i revert these changes and do amplify status, these items are always removed and I don't see the new Lambda on amplify status listing.

parvusville commented 1 year ago

One more thing I noticed after resetting staging to previous state and trying to do the merge again. After merge I do amplify push and enter the missing env variables and secrets, and then at ? Are you sure you want to continue? (Y/n) ‣ I observe changes done by amplify. There are same things as listed above in amplify-dependent-resources-ref.d.ts, function-parameters.json and app.js, but now in **backend-config.json** I see the whole "parameters" section being removed. Is this section supposed to be there or not, and what does it actually do?

What is the way to proceed here? Currently since there seems to be so many problems at the same time (missing env variables, missing secrets and a missing function) I'm thinking of reverting everything in dev and adding them back one by one..

josefaidt commented 1 year ago

Hey @parvusville :wave: thanks for raising this! Reading the description, this appears similar to https://github.com/aws-amplify/amplify-cli/issues/8090, where you will need to run amplify env checkout <current-env-name> after git merge

# on amplify env "dev", git branch "dev"
git checkout prod
amplify env checkout prod
git merge dev
amplify env checkout prod
amplify push

Can you see if this resolves the original issue?

parvusville commented 1 year ago

Hello @josefaidt . Doing this now shows the new function on "Create" status in staging. However when I do the push, like said the CLI automatically removes the whole "parameters" section from backend-config.json like before. At this point

✔ Are you sure you want to continue? (Y/n)

I have not proceeded with the push, since I don't know if those values should be there or not, and what might happen if I push without them. This now seems to happen even if I remove the lambda in dev and merge that into staging.

josefaidt commented 1 year ago

Hey @parvusville thanks for clarifying. I do see where the s3key and deploymentBucketName parameters are removed from amplify/backend/backend-config.json when we start the push, however they will reappear after pushing.

I also see the issue where we add a secret to a Function in one environment, merge it over, and the CLI shows "no change" for the function. After making an arbitrary change to the Function's source code and pushing I see the new secret added:

here's a few steps I captured in the reproduction process:

after merging the newly created "next" environment into "dev" where the secret was added https://github.com/josefaidt/a12188/commit/28ba2920886dfcc19101d8b64921145e897f6360
the arbitrary change https://github.com/josefaidt/a12188/commit/4c24259ff6ac17cf8ebf3d2e23dfe038a03f2332
"pre-push" state as I was pushing to the "dev" env https://github.com/josefaidt/a12188/commit/fa102e8bf867b0010d0f2f0bc28de6fff0bfbae4
- same for "next" https://github.com/josefaidt/a12188/commit/fa102e8bf867b0010d0f2f0bc28de6fff0bfbae4
"post-push" state as the push completed https://github.com/josefaidt/a12188/commit/d816df7fb20eb3e6237728a1fab00a2aed028687
- same for "next" https://github.com/josefaidt/a12188/commit/d816df7fb20eb3e6237728a1fab00a2aed028687

From the git diffs above, are these the changes you're seeing to your backend-config?

josefaidt commented 1 year ago

Additionally, I see the new secret is not created in SSM for the target environment "dev"

To workaround this, run amplify update function and choose to modify secrets configuration, then update the new secret to trigger a change to SSM Parameter Store

➜  amplify update function
? Select the Lambda function you want to update a121889f9a3b3e
General information
- Name: a121889f9a3b3e
- Runtime: nodejs

Resource access permission
- Not configured

Scheduled recurring invocation
- Not configured

Lambda layers
- Not configured

Environment variables:
- Not configured

Secrets configuration
- mysecret
- myothersecret
- anewsecret

? Which setting do you want to update? Secret values configuration
? What do you want to do? Update a secret
? Select the secret to update: anewsecret
? Enter the value for anewsecret: [hidden]
? What do you want to do? I'm done
? This will immediately update secret values in the cloud. Do you want to continue? Yes
Use the AWS SSM GetParameter API to retrieve secrets in your Lambda function.
More information can be found here: https://docs.aws.amazon.com/systems-manager/latest/APIReference/API_GetParameter.html
? Do you want to edit the local lambda function now? No

parvusville commented 1 year ago

Hello @josefaidt . Those backend-config changes are probably similar to what I originally witnessed. Now however as told the problem is the whole parameters section is removed when I try to do amplify push. After the original post I did remove (in dev before merging to staging) variables, secrets and the function that were causing trouble when merging to staging, to try to add them back one by one.

First I tried with just adding the plain function with nothing added (in dev), but as previously explained I'm now facing the situation where all parameters are removed when i merge to staging and do amplify push. Even without the new Lambda function from dev. I mean the whole "parameters": {...} block at backend-config. I see this in git diff when I'm about to push, but I have not proceeded when CLI asks ✔ Are you sure you want to continue? (Y/n).

I don't know if or how this is connected to the original problems I described, but this is what is blocking me right now.

parvusville commented 1 year ago

Hey, just as an update I have now successfully completed amplify push. @josefaidt . Like you witnessed before, apparently the parameters block re-appears after pushing. Before at your comment you just mentioned certain keys being removed from parameters and then re-added after push, and that left me worried as I noticed the whole parameters block being removed.

So for now I seem to be able to merge and push changes with the workarounds from previous messages, related to adding a new function and secrets.

josefaidt commented 1 year ago

Hey @parvusville thanks for clarifying! I'm glad to hear you're able to successfully push despite the oddity! We'll keep this labeled as a bug for the team to evaluate further

goldbez commented 1 year ago

This issue was fixed in version 12.0.0.

github-actions[bot] commented 1 year ago

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

aws-amplify / amplify-cli