patchbio / notion-tools

Python tools to work with Notion
Apache License 2.0
8 stars 3 forks source link

Accommodate variable rate limits #2

Open heuermh opened 2 years ago

heuermh commented 2 years ago

To work around Notion request limits, this library uses a static delay

    while response["has_more"]:
        sleep(0.1)
        response = notion_client.databases.query(...

It would be desirable to accommodate variable rate limits as described in the Notion doc, copied here:

Rate limits Rate-limited requests will return a "rate_limited" error code (HTTP response status 429). The rate limit for incoming requests is an average of 3 requests per second. Some bursts beyond the average rate are allowed.

Integrations should accommodate variable rate limits by handling HTTP 429 responses and respecting the Retry-After response header value, which is set as an integer number of seconds (in decimal). Requests made after waiting this minimum amount of time should no longer be rate limited.

Alternatively, rate limits can be accommodated by backing off (or slowing down) the speed of future requests. A common way to implement this is using one or several queues for pending requests, which can be consumed by sending requests as long as Notion does not respond with an HTTP 429.