Latest GYB and QueriesPerMinutePerUser rate limits not caught

[mf]

GYB (1.42, Ubuntu) running with Service Account (GSuite), checked the limits (15,000 per minute at max) Large email box (~4m messages)

Receiving this output hammering stdout:

'Queries per minute per user' of service 'gmail.googleapis.com' for consumer 'project_number:XXXXXXXXXXXX'."> <HttpError 403 when requesting https://gmail.googleapis.com/gmail/v1/users/me/messages/1673611fd07e15f5?format=raw&fields=id%2ClabelIds%2CinternalDate%2Craw&alt=json returned "Quota exceeded for quota metric 'Queries' and limit 'Queries per minute per user' of service 'gmail.googleapis.com' for consumer 'project_number:XXXXXXXXXXXX'.". Details: "Quota exceeded for quota metric 'Queries' and limit 'Queries per minute per user' of service 'gmail.googleapis.com' for consumer 'project_number:XXXXXXXXXXXX'.">

GYB is not detecting this as rate error, and continues to rapidly retrieve messages, as the result backup never finishes. Could it be that the limits are exhausted during retrieval of message IDs even before we start downloading?

[jay0lee]

blurg, Google really shouldn't be returning 403 errors for quota issues.

For these errors, what does the value of reason look like from here:

https://github.com/jay0lee/got-your-back/blob/main/gyb.py#L720

If it's something unique we can add it to the reasons to retry.

[mf]

The message is "Quota exceeded for quota metric 'Queries' and limit 'Queries per minute per user' of service ....", which I believe matches the "Queries per minute per user" == "userRateLimitExceeded"

There are only 3 enforced limits for the GSuite APIs (default values supplied):

[jay0lee]

so then adding userRateLimitExceeded to https://github.com/jay0lee/got-your-back/blob/main/gyb.py#L723 should retry these errors.

Jay Lee

On Mon, Mar 22, 2021 at 11:48 AM Michael Firsikov @.***> wrote:

The message is "Quota exceeded for quota metric 'Queries' and limit 'Queries per minute per user' of service ....", which I believe matches the "Queries per minute per user" == "userRateLimitExceeded"

There are only 3 enforced limits for the GSuite APIs (default values supplied): [image: image] https://user-images.githubusercontent.com/57658/112017862-6674e900-8b04-11eb-8177-fd49c54ee40d.png

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jay0lee/got-your-back/issues/309#issuecomment-804174021, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDIZMEXRWYFZJYWN2G5HBTTE5RD7ANCNFSM4ZTEAMHA .

[mf]

Also, does it make sense to add a possible run time option to "ratelimit" GYB from the user side? This would allow us to artificially ratelimit GYB based on some arbitrary number (X amount of queries per minute (or Y seconds))

[jay0lee]

I'd generally prefer not to. Google APIs are generally designed with the idea that you should be able to hit them as fast as your client code possibly can and only need to respond to errors with exponential backoff/retry.

Jay Lee

On Mon, Mar 22, 2021 at 11:50 AM Michael Firsikov @.***> wrote:

Also, does it make sense to add a possible run time option to "ratelimit" GYB from the user side? This would allow us to artificially ratelimit GYB based on some arbitrary number (X amount of queries per minute (or Y seconds))

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jay0lee/got-your-back/issues/309#issuecomment-804176135, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDIZMDMELQEX3CWMNCGEZLTE5RM7ANCNFSM4ZTEAMHA .

[wwc559]

Is there a workaround for this problem? It is making it impossible for me to backup my (admittedly large) gmail account.

This seems a wonderful package you are developing, Thanks!

Bill Carlson

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.