search

aws-amplify / amplify-category-api

The AWS Amplify CLI is a toolchain for simplifying serverless web and mobile development. This plugin provides functionality for the API category, allowing for the creation and management of GraphQL and REST based backends for your amplify project.
https://docs.amplify.aws/
Apache License 2.0
90 stars 76 forks source link

Can't update my API key while the old one expired #636

Open fly1030 opened 2 years ago

fly1030 commented 2 years ago

Before opening, please confirm:

How did you install the Amplify CLI?

npm

If applicable, what version of Node.js are you using?

16.13.2

Amplify CLI Version

9.1.0

What operating system are you using?

Mac

Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.

Amplify Categories

api

Amplify Commands

push

Describe the bug

Part of the issue here is in https://github.com/aws-amplify/amplify-category-api/issues/598, where I also commented

Expected behavior

Expect an easier way to update API key used in deployed version

Reproduction steps

Refer to description, there are exact steps

GraphQL schema(s)

```graphql # Put schemas below this line ```

Project Identifier

No response

Log output

``` # Put your logs below this line ```

Additional information

No response

fly1030 commented 2 years ago

Any chance to take a look at this? I'm still blocked on this

josefaidt commented 2 years ago

Hey @fly1030 would you mind emailing us your Account ID and API ID to amplify-cli@amazon.com (with a reference to this issue number)? I'd like to work with our AppSync team to see if we can find a suitable mitigation for you. While you are doing that I will work to see if I can find a suitable workaround from a reproduction standpoint.

fly1030 commented 2 years ago

Sent required information in email

fly1030 commented 2 years ago

@josefaidt any chance there's something I can try? We're still in limbo state.

fly1030 commented 2 years ago

following up again, any updates?

fly1030 commented 2 years ago

Alright, so we couldn't wait anymore so I went ahead and did 'amplify api remove', then pushed and started from scratch again with amplify add api... It fixed the key problem, but all data is gone. We'll recover the data somehow, but guess that's better than having everything down. Still interested in knowing WA for future references.

josefaidt commented 2 years ago

Hey @fly1030 apologies for the delay here, while the team continues to improve the experience if this arises again please remove references to GraphQLAPIKeyOutput in the backend-config.json file and set CreateAPIKey to 0 as you've noted in your original post. This should allow us to push and delete the API key while not disturbing the function push. From there we can revert our changes to backend-config.json and create another API key with CreateAPIKey: 1

josefaidt commented 2 years ago

I've also marked this as a bug to improve the experience where we have resources dependent on the API key output, which can ultimately be mitigated by removing the two-step process of rotating the API key in favor of a single command. This behavior is documented as a feature request here https://github.com/aws-amplify/amplify-category-api/issues/598

ejmiller2 commented 2 years ago

@josefaidt I am having a very similar problem. My API Key expired. I used AppSync console to create a new key (da2-NEWKEY) and then deleted the old key (da2-OLDKEY). My web (React) application and 2 Lambda functions can no longer access the data behind the API because they use the old key.

I tried to deploy changes in Amplify Studio but it failed with a message like:

Deployment failed 10/11/2022, 10:18:30 PM: API key not found: da2-OLDKEY

In Amplify Studio under Data modeling -> Manage API authorization mode, I see the new key (da2-NEWKEY), but if I try to Save & deploy, I get an error like above and everything reverts. I tried to switch to a different authorization method (e. g. IAM), but again it fails due to the old keys.

Running amplify status on my development computer also lists the old key: GraphQL API KEY: da2-OLDKEY If I try to push changes from my development computer, they fail due to the missing old key and reverts. I have tried setting "CreateAPIKey": 0 in parameters.json, but it fails (I think) due to the Lambdas. I tried to remove key references from the Lambdas and backend-config.json file, but it still failed. I this case I get the following in Amplify Studio for each of the Lambdas:

Output 'GraphQLAPIKeyOutput' not found in stack

I got the 2 Lambda functions working by manually changing the MYAPP_GRAPHQLAPIKEYOUTPUT in Configuration -> Environment variables.

I can get my local application working by manually changing aws_appsync_apiKey in aws-exports.js (of course this reverts if I do an amplify pull).

How can I get my deployed application running again?

I can't afford to lose the data, I have a presentation on it at re:Invent which I need to complete in a week or so. :(

I think the previous time my API key expired, I just extended it in Amplify Studio. What is the best practice for rotating keys? Or should I be using Cognito or IAM?

ejmiller2 commented 1 year ago

In my case, the old key was deleted, not just expired. Should this be a separate issue? Also, I believe other aspects of the stack, such as Lambdas which also use the API keys, make this more complicated.

sammyiyke commented 1 year ago

I am currently facing the same challenge with deleted, expired keys. I also have lambdas that are dependent on the keys too, which makes the entire process messy.

ejmiller2 commented 1 year ago

@josefaidt @sammyiyke I finally got my application working again, starting with the info in #aws-amplify/amplify-cli in this comment. I did not use APIKeyExpirationEpoch, as this seems obsolete. I followed the first 4 steps successfully, but steps 5-8 failed to push, due to the key not being available for the Lambdas. So I skipped step 5 (I left out the references to GraphQLAPIKeyOutput) but did step six (removing CreateAPIKey: 0) followed by 7 and 8. This time the push worked! Now my web application worked, but the Lambdas were not using the API KEY at all, so they still don't work. I tried to redo step 5 followed by 7 and 8, but the push did nothing. I ended up using amplify update function to remove the API from the resources for each Lambda, then used the same function to add them back in. This time the push worked, and the application was working!

Here's a summary of what (I think) worked: Note: I upgraded the amplify CLI to version 10.3.1. I don't know if this mattered.

  1. In api/[name]/parameters.json, set CreateAPIKey: 0 (be sure to add the comma after the previous line if you are dding this at the end).
  2. In backend/backend-config.json remove any JSON attribute array values of GraphQLAPIKeyOutput. For example, change the following (there should be one for each Lambda which uses the API): 
          "attributes": [
          "GraphQLAPIIdOutput",
          "GraphQLAPIEndpointOutput",
          "GraphQLAPIKeyOutput"
      ]

    to

          "attributes": [
          "GraphQLAPIIdOutput",
          "GraphQLAPIEndpointOutput"
      ]
  3. amplify env checkout [envName]
  4. amplify push -y - on completion, the key should have been removed from the application and Lambdas.
  5. In api/[name]/parameters.json, remove CreateAPIKey: 0
  6. amplify env checkout [envName]
  7. amplify push -y- on completion, the key should have been restored to the application, but not the Lambdas
  8. Use amplify update function to remove the API in question from the resources of each Lambda.
  9. Use amplify update function to restore the API in question from the resources of each Lambda.
  10. amplify push -y - on completion, everything is working! (at least it was for me)
fly1030 commented 1 year ago

@josefaidt @sammyiyke I finally got my application working again, starting with the info in #aws-amplify/amplify-cli in this comment. I did not use APIKeyExpirationEpoch, as this seems obsolete. I followed the first 4 steps successfully, but steps 5-8 failed to push, due to the key not being available for the Lambdas. So I skipped step 5 (I left out the references to GraphQLAPIKeyOutput) but did step six (removing CreateAPIKey: 0) followed by 7 and 8. This time the push worked! Now my web application worked, but the Lambdas were not using the API KEY at all, so they still don't work. I tried to redo step 5 followed by 7 and 8, but the push did nothing. I ended up using amplify update function to remove the API from the resources for each Lambda, then used the same function to add them back in. This time the push worked, and the application was working!

Here's a summary of what (I think) worked: Note: I upgraded the amplify CLI to version 10.3.1. I don't know if this mattered.

  1. In api/[name]/parameters.json, set CreateAPIKey: 0 (be sure to add the comma after the previous line if you are dding this at the end).
  2. In backend/backend-config.json remove any JSON attribute array values of GraphQLAPIKeyOutput. For example, change the following (there should be one for each Lambda which uses the API):
          "attributes": [
              "GraphQLAPIIdOutput",
              "GraphQLAPIEndpointOutput",
              "GraphQLAPIKeyOutput"
          ]

to

          "attributes": [
              "GraphQLAPIIdOutput",
              "GraphQLAPIEndpointOutput"
          ]
  1. amplify env checkout [envName]
  2. amplify push -y - on completion, the key should have been removed from the application and Lambdas.
  3. In api/[name]/parameters.json, remove CreateAPIKey: 0
  4. amplify env checkout [envName]
  5. amplify push -y- on completion, the key should have been restored to the application, but not the Lambdas
  6. Use amplify update function to remove the API in question from the resources of each Lambda.
  7. Use amplify update function to restore the API in question from the resources of each Lambda.
  8. amplify push -y - on completion, everything is working! (at least it was for me)

Thanks for sharing Ed, very useful information for later reference!

parvusville commented 1 year ago

I'm also facing this issue with deleted API keys. Trying what @josefaidt suggested and @ejmiller2 demonstrated above did not work for me.

I tried with both 

"CreateApiKey": 0

and 

  "CreateApiKey": 0,
  "APIKeyExpirationEpoch": -1

while having GraphQLAPIKeyOutput references removed from the backend-config.json. Pushing still fails with 

🛑 The following resources failed to deploy:
Resource Name: GraphQLAPIDefaultApiKey215A6DD7 (AWS::AppSync::ApiKey)
Event Type: update
Reason: API key not found: da2-6j62dzthqvcuph6bwokehv6nda (Service: AWSAppSync; Status Code: 404; Error Code: NotFoundException; Request ID: 9a606262-f01c-4ffb-a115-8a905b1420c5; Proxy: null)

Any suggestions on how to work around this?

bstascavage commented 1 year ago

The fact that there is no easy way to update an expired API key and that the fix took me HOURS to do is beyond stupid. I can't believe a team of engineers can see a ticket like this and say "Yup, we designed this well."

Its an API key; updating it when it expired is one of the most trivial operations an engineer can do. So thank you for making me jump through 30 hoops to do so :roll_eyes:

duranmla commented 1 year ago

Judging from the answers I think we just needed to say to amplify "here we are again" and using the CLI to push a dumb change made the trick for me. What I have done is to:

  1. Add a change into the schema "an attr to a model"
  2. push the change
  3. Now a new API Key has been created for me

AppSync console will be like:

Before:

image

After:

image

Not super hard after all. ❤️

Sanoodia commented 1 year ago

Im facing issue related to this ticket, that i have updated key on cloud in AppSync api but when i pushed it from amplify-cli its give me this error.

Resource Name: GraphQLAPIDefaultApiKey215A6DD7 (AWS::AppSync::ApiKey) Event Type: update Reason: API key not found: da2-6pw4pntar5ft5k6o6qq3d5b6ma (Service: AWSAppSync; Status Code: 404; Error Code: NotFoundException; Request ID: 599c43cd-5475-41d7-a026-31487a9eed88; Proxy: null)

and this one is my old key da2-6pw4pntar5ft5k6o6qq3d5b6ma the error is mentioning. what should i do, need help.

duranmla commented 1 year ago

Today I face this issue again and https://github.com/aws-amplify/amplify-category-api/issues/636#issuecomment-1288164584 this solved it. One thing it wasn't clear for me was that when he mentioned amplify env checkout [envName] I didn't known what he wanted exactly but reading the comment he pointed it seems we need to checkout env back and forward. So every step of amplify env checkout [envName] for me was going to prod and then back to my env

gringrape commented 1 year ago

@josefaidt @sammyiyke I finally got my application working again, starting with the info in #aws-amplify/amplify-cli in this comment. I did not use APIKeyExpirationEpoch, as this seems obsolete. I followed the first 4 steps successfully, but steps 5-8 failed to push, due to the key not being available for the Lambdas. So I skipped step 5 (I left out the references to GraphQLAPIKeyOutput) but did step six (removing CreateAPIKey: 0) followed by 7 and 8. This time the push worked! Now my web application worked, but the Lambdas were not using the API KEY at all, so they still don't work. I tried to redo step 5 followed by 7 and 8, but the push did nothing. I ended up using amplify update function to remove the API from the resources for each Lambda, then used the same function to add them back in. This time the push worked, and the application was working!

Here's a summary of what (I think) worked: Note: I upgraded the amplify CLI to version 10.3.1. I don't know if this mattered.

  1. In api/[name]/parameters.json, set CreateAPIKey: 0 (be sure to add the comma after the previous line if you are dding this at the end).
  2. In backend/backend-config.json remove any JSON attribute array values of GraphQLAPIKeyOutput. For example, change the following (there should be one for each Lambda which uses the API):
          "attributes": [
              "GraphQLAPIIdOutput",
              "GraphQLAPIEndpointOutput",
              "GraphQLAPIKeyOutput"
          ]

to

          "attributes": [
              "GraphQLAPIIdOutput",
              "GraphQLAPIEndpointOutput"
          ]
  1. amplify env checkout [envName]
  2. amplify push -y - on completion, the key should have been removed from the application and Lambdas.
  3. In api/[name]/parameters.json, remove CreateAPIKey: 0
  4. amplify env checkout [envName]
  5. amplify push -y- on completion, the key should have been restored to the application, but not the Lambdas
  6. Use amplify update function to remove the API in question from the resources of each Lambda.
  7. Use amplify update function to restore the API in question from the resources of each Lambda.
  8. amplify push -y - on completion, everything is working! (at least it was for me)

Thank you!

Howwwwwwwwww do you discover this procedure?

Thank you!!! you re my life savior

duranmla commented 1 year ago

Me again doing some updates. I have face this issue several times (As I have been on and off working on this project for few months now). The best comment is the one from ejmiller2 which will help us to have the system back and running, however, there are cases where you still have API Key access with 401 errors cause API Key doesn't get updated, if you go AppSync console > settings > API Keys and you see yours expired, you need to:

After doing ejmiller2 comment, push a dummy update on the schema to trigger an API Key update. Otherwise, whenever you have a model with API Key access (for guest unauthorised users) it will fail

olliethedev commented 12 months ago

Simply adding CreateAPIKey: 0 then pushing then removing CreateAPIKey: 0 from api/[name]/parameters.json has worked for me.

Really hope this issue is fixed eventually because this is probably my fifth time needing to do this over the last 2 years.

irfanmurtaza-el commented 11 months ago

I've similar issue, but my Stack is in UPDATE_ROLLBACK_FAILED, this is not allowing me to push to test any of above approach. When I tried to continue update rollback from AWS console in cloud-formation, it doesn't succeeded. The stack is failing due to API key must be valid for a minimum of 1 days. (Service: AWSAppSync; Status Code: 400; Error Code) which is understood as previously stack has default key expiration set to 28th Sep, that's why stack is even not rolling back to previous state. Any help regarding this? Is there any way I can update it's expiry key to somewhat newer one?

phreitom commented 11 months ago

I've similar issue, but my Stack is in UPDATE_ROLLBACK_FAILED, this is not allowing me to push to test any of above approach. When I tried to continue update rollback from AWS console in cloud-formation, it doesn't succeeded. The stack is failing due to API key must be valid for a minimum of 1 days. (Service: AWSAppSync; Status Code: 400; Error Code) which is understood as previously stack has default key expiration set to 28th Sep, that's why stack is even not rolling back to previous state. Any help regarding this? Is there any way I can update it's expiry key to somewhat newer one?

I am getting the exact same issue...dying.

chadpatel commented 8 months ago

This is still an issue. I have an application I use for ~1 week a year and every year I spend a ton of time fixing broken crap like API keys :| I feel like this should just work

gyamini25 commented 8 months ago

Here's what worked for me: dive into your parameters.json file nestled snugly within your amplify folder, specifically at amplify/backend/api/yourprojectname/parameters.json.

Tweak that file by adding in "CreateAPIKey": 0, then execute amplify push, kick back while the magic happens. Once that's done, crank "CreateAPIKey": 1, and repeat with amplify push, letting the gears turn.

Next up, jazz things up with your very own custom APIKeyExpirationEpoch. Just slot in "APIKeyExpirationEpoch": XXXXXXXXX, hit up amplify push one last time, and sit tight for the final transformation.

Boom, you're golden! Give amplify status a quick click to double-check your handiwork!

chrislrobert commented 8 months ago

After many many many frustrating hours trying everything I could find to resolve the issue described here, I finally got my env back up by following a modified version of the solution here:

  1. Merge into branch in source repo, triggering new build in Amplify
  2. Observe that Amplify build fails with “API key not found” error
  3. Go to AWS plugin within PyCharm, update credentials, open local console
  4. A first time: amplify env checkout BRANCH
  5. Edit amplify/backend/api/appname/parameters.json to add a CreateAPIKey: 0 parameter
  6. Edit amplify/backend/backend-config.json and remove all GraphQLAPIKeyOutput references
  7. A second time: amplify env checkout BRANCH
  8. A first time: amplify push -y
  9. Roll back all local changes (namely: steps 4 and 5 above)
  10. A third time: amplify env checkout BRANCH
  11. A second time: amplify push -y
  12. Go back to failed build in Amplify UI, click to “Redeploy this version” to rebuild

I had tried and failed with the same procedure previously, because I hadn't realized that the repeated checkouts were necessary (I skipped them because I had already checked out the env). This whole experience has been really frustrating, and it makes me regret having chosen Amplify for my app framework.

chrislrobert commented 8 months ago

However: after all this, I went back to Amplify Studio, and none of the "Manage content" functionality was working. The drop-down has my tables, but they show no content in the tables and I can't add records. I've tried:

  1. Disabling and re-enabling Amplify Studio to redeploy.
  2. Checking the GraphQL API settings in the Studio (they have the new/proper API key).
  3. Checking the GraphQL API connection (it points to the correct AppSync API, which points to the correct DynamoDB tables).
  4. Checking the DynamoDB tables (they still have all of their proper content).

This issue is true now in all four of my environments across two separate AWS accounts. While it's possible that the failure is unrelated to this issue here, everything was working a few days ago and all I've done since was wrestle with getting these API keys updated.

chrislrobert commented 8 months ago

This is what fixed my Amplify Studio Data Manager:

  1. amplify env checkout BRANCH
  2. amplify api gql-compile --force
  3. amplify push -y

Honestly, this whole experience has been a nightmare — and I see that, even after I edit my keys to be valid for longer, new deployments reset to 30 days. So I guess I have to manually deploy or extend the keys every 30 days, otherwise I face this nightmare again and again. It's a completely baffling design for an app framework that's intended for production apps.

curtismorte commented 6 months ago

To everyone in this thread, you can simply update the expiration date for your expired keys if you aren't rotating them.

Go to your AppSync API in the console > settings > edit api key > adjust the expiration date. Expiration dates can be as far as 365 days in the future.

chrislrobert commented 6 months ago

To everyone in this thread, you can simply update the expiration date for your expired keys if you aren't rotating them.

Go to your AppSync API in the console > settings > edit api key > adjust the expiration date. Expiration dates can be as far as 365 days in the future.

@curtismorte, the trouble is that the API keys are no longer there to adjust once they have expired. Or, if they are for some period post-expiration, they certainly were no longer there by the time I got to mine — and others seem to have had a similar issue. Once the keys are gone, you're well and truly in trouble, and you begin the nightmarish sequences to try to recover your app.

IAmBrendanL commented 5 months ago

I'll second what @chrislrobert said. Our production environment is fine, but our staging env hadn't been used in a awhile and the API key lapsed in that environment. I'm weary of following the steps in this thread as the documentation for amplify push doesn't specify if it's env specific or not. I do not want to rotate our production api key.

As a work-around I've been manually changing the API key in the config files to one I generated in the AppSync dashboard.

CameronWard301 commented 4 months ago

@josefaidt @sammyiyke I finally got my application working again, starting with the info in #aws-amplify/amplify-cli in this comment. I did not use APIKeyExpirationEpoch, as this seems obsolete. I followed the first 4 steps successfully, but steps 5-8 failed to push, due to the key not being available for the Lambdas. So I skipped step 5 (I left out the references to GraphQLAPIKeyOutput) but did step six (removing CreateAPIKey: 0) followed by 7 and 8. This time the push worked! Now my web application worked, but the Lambdas were not using the API KEY at all, so they still don't work. I tried to redo step 5 followed by 7 and 8, but the push did nothing. I ended up using amplify update function to remove the API from the resources for each Lambda, then used the same function to add them back in. This time the push worked, and the application was working!

Here's a summary of what (I think) worked: Note: I upgraded the amplify CLI to version 10.3.1. I don't know if this mattered.

  1. In api/[name]/parameters.json, set CreateAPIKey: 0 (be sure to add the comma after the previous line if you are dding this at the end).
  2. In backend/backend-config.json remove any JSON attribute array values of GraphQLAPIKeyOutput. For example, change the following (there should be one for each Lambda which uses the API):
          "attributes": [
              "GraphQLAPIIdOutput",
              "GraphQLAPIEndpointOutput",
              "GraphQLAPIKeyOutput"
          ]

to

          "attributes": [
              "GraphQLAPIIdOutput",
              "GraphQLAPIEndpointOutput"
          ]
  1. amplify env checkout [envName]
  2. amplify push -y - on completion, the key should have been removed from the application and Lambdas.
  3. In api/[name]/parameters.json, remove CreateAPIKey: 0
  4. amplify env checkout [envName]
  5. amplify push -y- on completion, the key should have been restored to the application, but not the Lambdas
  6. Use amplify update function to remove the API in question from the resources of each Lambda.
  7. Use amplify update function to restore the API in question from the resources of each Lambda.
  8. amplify push -y - on completion, everything is working! (at least it was for me)

I had to do this process today. I did steps 1, 2, 4, 5, 7 and it worked for me.

squirrelhomie commented 2 months ago

I'm having a similar issue. My stack is in UPDATE_ROLLBACK_COMPLETE and when I run amplify push I get this error - DeploymentFault: Resource is not in the state stackUpdateComplete

At this point, I can't remember what I did exactly. I believe the api key was expired and at some point, deleted the api key in app sync, created a new one.

I've tried CreateAPIKey: 0 with no helpful errors following. When I add APIKeyExpirationEpoch: -1 to my parameters.json I get a somewhat more helpful error:

Resource Name: GraphQLAPIDefaultApiKey<key> (AWS::AppSync::ApiKey)
Event Type: update
Reason: API key not found: <api-key> (Service: AWSAppSync; Status Code: 404; Error Code: NotFoundException; Request ID: ; Proxy: null)

The api key it is referencing is one that is found in my aws-exports but I've since deleted that key from app sync.

At this point, not sure what direction to go. Any help would be appreciated!

squirrelhomie commented 2 months ago

I'm also facing this issue with deleted API keys. Trying what @josefaidt suggested and @ejmiller2 demonstrated above did not work for me.

I tried with both

"CreateApiKey": 0

and

  "CreateApiKey": 0,
  "APIKeyExpirationEpoch": -1

while having GraphQLAPIKeyOutput references removed from the backend-config.json. Pushing still fails with

🛑 The following resources failed to deploy:
Resource Name: GraphQLAPIDefaultApiKey215A6DD7 (AWS::AppSync::ApiKey)
Event Type: update
Reason: API key not found: da2-6j62dzthqvcuph6bwokehv6nda (Service: AWSAppSync; Status Code: 404; Error Code: NotFoundException; Request ID: 9a606262-f01c-4ffb-a115-8a905b1420c5; Proxy: null)

Any suggestions on how to work around this?

@parvusville I'm having the same issue. CreateAPIKey: 0 is not working for me and I am getting the same error you mentioned. How were you able to resolve this?

KeitaIsFree commented 1 month ago

I found that although it seems redundant to repeatedly amplify env checkout [envName], this is actually necessary. Don't skip this.

adriaanbalt commented 1 day ago

I found some documentation regarding CreateAPIKey here: https://docs.amplify.aws/gen1/react/tools/cli-legacy/config-params/