search

mifi / cognito-backup

CLI for backing up AWS Cogntito User Pools
MIT License
154 stars 26 forks source link

TooManyRequestsException: Too many requests #28

Closed hykw closed 2 years ago

hykw commented 2 years ago

I used to be able to dump normally, but since June 14th, 15:01 (UTC), I've got the following error message and the process stops halfway through.

I have two Cognito pools, each in a different AWS account, but I get the same error on both of them.

Do you know what is causing this?

TooManyRequestsException: Too many requests
    at Request.extractError (/home/ec2-user/.asdf/installs/nodejs/17.9.0/.npm/lib/node_modules/cognito-backup/node_modules/aws-sdk/lib/protocol/json.js:52:27)
    at Request.callListeners (/home/ec2-user/.asdf/installs/nodejs/17.9.0/.npm/lib/node_modules/cognito-backup/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/home/ec2-user/.asdf/installs/nodejs/17.9.0/.npm/lib/node_modules/cognito-backup/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/home/ec2-user/.asdf/installs/nodejs/17.9.0/.npm/lib/node_modules/cognito-backup/node_modules/aws-sdk/lib/request.js:686:14)
    at Request.transition (/home/ec2-user/.asdf/installs/nodejs/17.9.0/.npm/lib/node_modules/cognito-backup/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/home/ec2-user/.asdf/installs/nodejs/17.9.0/.npm/lib/node_modules/cognito-backup/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /home/ec2-user/.asdf/installs/nodejs/17.9.0/.npm/lib/node_modules/cognito-backup/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/home/ec2-user/.asdf/installs/nodejs/17.9.0/.npm/lib/node_modules/cognito-backup/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/home/ec2-user/.asdf/installs/nodejs/17.9.0/.npm/lib/node_modules/cognito-backup/node_modules/aws-sdk/lib/request.js:688:12)
    at Request.callListeners (/home/ec2-user/.asdf/installs/nodejs/17.9.0/.npm/lib/node_modules/cognito-backup/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
mifi commented 2 years ago

did your data (number of users) increase over the last days? Are you running cognito-backup backup-users only? I think this could be a bug in aws-sdk, because it's supposed to handle retry with back-off automatically

hykw commented 2 years ago

A couple of users increased then, but the total number is only 20,000.

I'm running backup-users only, some parameters are specified tough: 

backup-users $POOL_ID --region ap-northeast-1 --profile $PROFILE --file backup/$TODAY.users.json

According to CHANGELOG.md, aws-sdk is 2.1125.0, which was installed with cognito-backup.

# Changelog for AWS SDK for JavaScript
<!--LATEST=2.1125.0-->
<!--ENTRYINSERT-->
mifi commented 2 years ago

AFAIK the AWS SDK should automatically retry on these kind of errors: https://docs.aws.amazon.com/sdkref/latest/guide/feature-retry-behavior.html

not sure why it's not doing it. you don't happen to have any minimal reproducible code you can share?

hykw commented 2 years ago

Hi! Through trial and error, I've found that half the limit worked fine.

c.f. https://github.com/hykw/cognito-backup/commit/4f0b5db5cc9c3593cfe0ef34b70a1ba8a8d9013b#diff-180b3d2af89416b99563fff8c37eb8ca1ef7591fc1ddd736b5dacf740ccd3760L78

mifi commented 2 years ago

Cool! I have now replaced the bottleneck module with aws-sdk v3's built in throttling logic, so it should in theory not require any trial and error and should automatically back-off if it gets throttled. try out the new version will you! (3.0.1)

hykw commented 2 years ago

@mifi Hi! Thank you for the update. Though the backup time took longer than before, 3.0.1 worked correctly!

FYI. I have about 46000 users in a pool.

mifi commented 2 years ago

thanks for testing this! do you have groups also that gets backed up? are most users connected to at least one group?

mifi commented 2 years ago

I have released a new version with a flag --concurrency. maybe you can try that when you have time (e.g. try set it to 2)

hykw commented 2 years ago

@mifi Hi! Significantly improved performance!

concurrency minutes
2 19 min
4 16 min

I have a few groups, but most of the users are connected to one of the group.

mifi commented 2 years ago

Ok, that's great. and no errors?

hykw commented 2 years ago

at least so far(until concurrency=4), no errors at all!