Console timeouts with large deployments

cubs818 commented 2 weeks ago

Environment information

System:
  OS: Linux 5.15 Ubuntu 22.04.4 LTS 22.04.4 LTS (Jammy Jellyfish)
  CPU: (8) x64 Intel(R) Xeon(R) E-2244G CPU @ 3.80GHz
  Memory: 4.50 GB / 7.69 GB
  Shell: /bin/bash
Binaries:
  Node: 18.19.0 - ~/.nvm/versions/node/v18.19.0/bin/node
  Yarn: undefined - undefined
  npm: 10.2.3 - ~/.nvm/versions/node/v18.19.0/bin/npm
  pnpm: undefined - undefined
NPM Packages:
  @aws-amplify/auth-construct: 1.1.6
  @aws-amplify/backend: 1.0.4
  @aws-amplify/backend-auth: 1.1.0
  @aws-amplify/backend-cli: 1.2.2
  @aws-amplify/backend-data: 1.1.0
  @aws-amplify/backend-deployer: 1.0.3
  @aws-amplify/backend-function: 1.2.0
  @aws-amplify/backend-output-schemas: 1.1.0
  @aws-amplify/backend-output-storage: 1.0.2
  @aws-amplify/backend-secret: 1.0.0
  @aws-amplify/backend-storage: 1.0.4
  @aws-amplify/cli-core: 1.1.1
  @aws-amplify/client-config: 1.1.2
  @aws-amplify/deployed-backend-client: 1.2.0
  @aws-amplify/form-generator: 1.0.0
  @aws-amplify/model-generator: 1.0.3
  @aws-amplify/platform-core: 1.0.4
  @aws-amplify/plugin-types: 1.1.0
  @aws-amplify/sandbox: 1.1.1
  @aws-amplify/schema-generator: 1.2.0
  aws-amplify: 6.4.3
  aws-cdk: 2.149.0
  aws-cdk-lib: 2.149.0
  typescript: 5.5.4
AWS environment variables:
  AWS_STS_REGIONAL_ENDPOINTS = regional
  AWS_NODEJS_CONNECTION_REUSE_ENABLED = 1
  AWS_SDK_LOAD_CONFIG = 1
No CDK environment variables

Describe the bug

Have a deployment that spins up various media service products. In one branch I have 3 ec2, 4 Medialive Channels, 16 MediaConnect flows, and then ~10 tables and few other aopi data sources. In another branch the only difference is its 41 Medialive Channels, and 192 MediaConnect Flows. In that branch the console Times out and gives a rate limit exceeded error. Wont show the deployed resources, can not interact with Data, Auth or Storage. Each of those go back to what looks an empty config. The app is performing well and I can access all the backed within their respective services just not the console.

Reproduction steps

I guess make a lot of resources? Not really sure what is triggering it other then that.

Jay2113 commented 1 week ago

Hi @cubs818 👋 , thanks for raising this! Can you share the following information?

When deploying these resources, does the deployment ultimately succeed or fail?
Can you capture a HAR file and share it with us? This will be helpful for us to investigate which requests fails with the rate limit exceeded exception. Steps to capture the HAR file are outlined here: https://repost.aws/knowledge-center/support-case-browser-har-file.
Is this an intermittent issue or are you completely blocked on viewing the deployed resources section in the console and accessing the Data, Auth and Storage tabs in the console?
Lastly, can you provide us access to a git repository with a minimal backend example that demonstrates this issue so we could replicate it on our end?

Jay2113 commented 2 days ago

@cubs818 Following up to see if you are still blocked by this issue?

cubs818 commented 2 days ago

Apologies for the delay. I was still blocked by the issue, and I was hoping to recreate it. Issue is that the event this was for has passed and we have removed all of the resources. So I would have to recreate it and I was hoping to do so with a more genric backend to share with you.

To answer 2 of your questions, the deployment succeded and I was able to access all the resources in their native sections of the console. Issue was just with the Amplify console.

It was not intermittent, everytime it would ultimately fail.

I will try and rouch something together though it was a lot of resources so I will need to open up quotas in my dev accounts to do testing.

Thanks for the response.

Craig

aws-amplify / amplify-backend