GAM-team / got-your-back

Got Your Back (GYB) is a command line tool for backing up your Gmail messages to your computer using Gmail's API over HTTPS.
https://github.com/GAM-team/got-your-back/wiki
Apache License 2.0
2.64k stars 209 forks source link

Latest GYB and QueriesPerMinutePerUser rate limits not caught #309

Closed mf closed 3 years ago

mf commented 3 years ago

GYB (1.42, Ubuntu) running with Service Account (GSuite), checked the limits (15,000 per minute at max) Large email box (~4m messages)

Receiving this output hammering stdout:

'Queries per minute per user' of service 'gmail.googleapis.com' for consumer 'project_number:XXXXXXXXXXXX'."> <HttpError 403 when requesting https://gmail.googleapis.com/gmail/v1/users/me/messages/1673611fd07e15f5?format=raw&fields=id%2ClabelIds%2CinternalDate%2Craw&alt=json returned "Quota exceeded for quota metric 'Queries' and limit 'Queries per minute per user' of service 'gmail.googleapis.com' for consumer 'project_number:XXXXXXXXXXXX'.". Details: "Quota exceeded for quota metric 'Queries' and limit 'Queries per minute per user' of service 'gmail.googleapis.com' for consumer 'project_number:XXXXXXXXXXXX'.">

GYB is not detecting this as rate error, and continues to rapidly retrieve messages, as the result backup never finishes. Could it be that the limits are exhausted during retrieval of message IDs even before we start downloading?

jay0lee commented 3 years ago

blurg, Google really shouldn't be returning 403 errors for quota issues.

For these errors, what does the value of reason look like from here:

https://github.com/jay0lee/got-your-back/blob/main/gyb.py#L720

If it's something unique we can add it to the reasons to retry.

mf commented 3 years ago

The message is "Quota exceeded for quota metric 'Queries' and limit 'Queries per minute per user' of service ....", which I believe matches the "Queries per minute per user" == "userRateLimitExceeded"

There are only 3 enforced limits for the GSuite APIs (default values supplied): image

jay0lee commented 3 years ago

so then adding userRateLimitExceeded to https://github.com/jay0lee/got-your-back/blob/main/gyb.py#L723 should retry these errors.

Jay Lee

On Mon, Mar 22, 2021 at 11:48 AM Michael Firsikov @.***> wrote:

The message is "Quota exceeded for quota metric 'Queries' and limit 'Queries per minute per user' of service ....", which I believe matches the "Queries per minute per user" == "userRateLimitExceeded"

There are only 3 enforced limits for the GSuite APIs (default values supplied): [image: image] https://user-images.githubusercontent.com/57658/112017862-6674e900-8b04-11eb-8177-fd49c54ee40d.png

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jay0lee/got-your-back/issues/309#issuecomment-804174021, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDIZMEXRWYFZJYWN2G5HBTTE5RD7ANCNFSM4ZTEAMHA .

mf commented 3 years ago

Also, does it make sense to add a possible run time option to "ratelimit" GYB from the user side? This would allow us to artificially ratelimit GYB based on some arbitrary number (X amount of queries per minute (or Y seconds))

jay0lee commented 3 years ago

I'd generally prefer not to. Google APIs are generally designed with the idea that you should be able to hit them as fast as your client code possibly can and only need to respond to errors with exponential backoff/retry.

Jay Lee

On Mon, Mar 22, 2021 at 11:50 AM Michael Firsikov @.***> wrote:

Also, does it make sense to add a possible run time option to "ratelimit" GYB from the user side? This would allow us to artificially ratelimit GYB based on some arbitrary number (X amount of queries per minute (or Y seconds))

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jay0lee/got-your-back/issues/309#issuecomment-804176135, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDIZMDMELQEX3CWMNCGEZLTE5RM7ANCNFSM4ZTEAMHA .

wwc559 commented 3 years ago

Is there a workaround for this problem? It is making it impossible for me to backup my (admittedly large) gmail account.

This seems a wonderful package you are developing, Thanks!

Bill Carlson

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.