search

gauteh / lieer

Fast email-fetching, sending, and two-way tag synchronization between notmuch and GMail
http://lieer.gaute.vetsj.com
Other
554 stars 62 forks source link

Hitting user quota for messages.get: missing (undocumented) 15k req/user/min limit #232

Open tadfisher opened 1 year ago

tadfisher commented 1 year ago

I am experiencing rate-limiting for the messages.get endpoint, which doesn't make much sense according to the documentation:

Per user rate limit 250 quota units per user per second, moving average (allows short bursts).

The messages.get unit is 5 requests, so 1250 requests/s should be fine, and the sync is not going anywhere near that rate.

So I created OAuth credentials to see if I could find what quota I'm hitting, and I see two pieces of evidence:

image

image

To me, this looks like an additional undocumented quota of 15k requests/user/minute, which I am obviously hitting. This explains the behavior I'm seeing as well, where the rate-limit errors start occurring about 1 minute into the sync.

So we may want to track a moving 1-minute window of requests and add additional delays to keep under this limit.

gauteh commented 1 year ago

Ah, yeah. That seems reasonable. If it does not cause the exponential delays, it should make a things work better in general. Mostly an issue with initial sync.

tor. 5. jan. 2023, 01:17 skrev Tad Fisher @.***>:

I am experiencing rate-limiting for the messages.get endpoint, which doesn't make much sense according to the documentation: Per user rate limit 250 quota units per user per second, moving average (allows short bursts).

The messages.get unit is 5 requests, so 1250 requests/s should be fine, and the sync is not going anywhere near that rate.

So I created OAuth credentials to see if I could find what quota I'm hitting, and I see two pieces of evidence:

[image: image] https://user-images.githubusercontent.com/129148/210673731-20c0460b-49e8-4d50-8987-1206e90569a3.png

[image: image] https://user-images.githubusercontent.com/129148/210673771-77a6a379-3b1d-4034-b70c-ea5cc7f7b489.png

To me, this looks like an additional undocumented quota of 15k requests/user/minute, which I am obviously hitting. This explains the behavior I'm seeing as well, where the rate-limit errors start occurring about 1 minute into the sync.

So we may want to track a moving 1-minute window of requests and add additional delays to keep under this limit.

— Reply to this email directly, view it on GitHub https://github.com/gauteh/lieer/issues/232, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAN364L44WGD54S4ABYUUDWQYHKVANCNFSM6AAAAAATRLNJSM . You are receiving this because you are subscribed to this thread.Message ID: @.***>