Rate limiting

[ivancsicsmarkus]

I have a Node.js app which tries to upload more then 30 records a second. Airtable API doesn't allow us to do that. But this way the retry logic dies after ~3 mins for me.

I've solved it for myself with RateLimiter

Currently it looks something like this:

const RateLimiter = require('limiter').RateLimiter;
const limiter = new RateLimiter(5, 'second');

// ...

limiter.removeTokens(1, () => base('TAB').create({ ... }, err => err ? console.log(err) : void 0));

What if we implemented this in airtable.js?

[Solinx]

I'd actually love to see AirTable implement a more robust rate limiting that allows for short bursts.

We're going to pull a data from tables to generate a few static JSON files that are going to be used by our application as data source. For a few seconds we're make a load of calls, and then for hours or even days it's going to be 0.

[EvanHahn]

@Solinx Thanks for the suggestion (and insight into your use case). We'll keep this in mind as we work on new versions of the API and of Airtable.js.

[coreyward]

Yeah I'd love to use Airtable as a source with Gatsby, but the rate limiting is rather restrictive and limiting. The way Gatsby works is to fetch an entire dataset, build a GraphQL schema using inferred types from that data, and then all queries (during development or build) run against the local GraphQL instance. So a flurry of requests, then nothing until another build. As it stands, requesting a handful of records can take several minutes due to rate-limit penalties.

While improving the rate limiting logic in this library would be nice, it's really just treating a symptom. The real issue is the petty 30-second penalty for exceeding 5rps (which a web browser does every single time it loads a website: there are typically 6 runner threads running). At the very least Airtable could adjust the rate limiting to operate over a larger interval (e.g. 50 requests over a rolling 10 second window). This would better accommodate short bursts of activity, and would probably force the whoever is tasked with the the work to reimplement this using a window so that a timeout penalty isn't even necessary.

[coreyward]

Ha, so if the same rate limiting rules were applied to the Airtable homepage (52 requests without trackers), it would take over 10 seconds to load at 5rps, without any penalty. With the 30-second penalty, assuming it is incurred on every 6th request, it would take 4 minutes and 48 seconds to load.

[EvanHahn]

We're aware that our rate limits are a problem for some users. We're working on some other features that may alleviate some of these concerns, and we'll let you know when those are released.

We care a lot about Airtable being a flexible toolkit, and the API is an important part of that. Unlike other services that are primarily backend databases (e.g. Firebase), we need to strike a balance between Airtable being a great experience for people using our apps interactively and API integrations.

By the way, we're going to try to funnel API feedback/questions away from Airtable.js and towards the API topic on our community forum. If there's an issue with Airtable.js, this is the place for it, but if it's about the API, we're going to try to move discussion there. Not going to close this issue or anything (especially because it partly deals with Airtable.js), but want to let folks know!