beyarkay
commented
1 year ago Heavily inspired by Cross-Ref's API: https://api.crossref.org/swagger-ui/index.html
Specifically, they send requests containing a mailto parameter to a dedicated pool of machines
snippet included below:
Etiquette
We want to provide a public, open, and free API for all. And we don't want to unnecessarily burden developers (or ourselves) with cumbersome API tokens or registration processes in order to use the public REST API. For that to work, we ask that you be polite and try not to do anything that will take the public REST API down or otherwise make it unusable for others. Specifically, we encourage the following polite behaviour:
- Cache data so you don't request the same data over and over again.
- Actively monitor API response times. If they start to go up, back-off for a while. For example, add pauses between requests and/or reduce the number of parallel requests.
- Specify a User-Agent header that properly identifies your script or tool and that provides a means of contacting you via email using "mailto:". For example: GroovyBib/1.1 (https://example.org/GroovyBib/; mailto:GroovyBib@example.org) BasedOnFunkyLib/1.4. This way we can contact you if we see a problem.
report problems and/or ask questions on our issue tracker.
Alas, not all people are polite. And for this reason we reserve the right to impose rate limits and/or to block clients that are disrupting the public service.
Good manners = more reliable service
But we prefer carrots to sticks. As of September 18th 2017 any API queries that use HTTPS and have appropriate contact information will be directed to a special pool of API machines that are reserved for polite users.
Why are we doing this? Well- we don't want to force users to have to register with us. But this means that if some user of the public server writes a buggy script or ignores timeouts and errors- they can really bring the API service to its knees. What's more, it is very hard for us to identify these problem users because they tend to work off multiple parallel machines and use generic User-Agent headers. They are effectively anonymous. We're starting to have to spend a lot of time dealing with these problems and the degraded performance of the public API is affecting all the polite users as well.
So... we are keeping the public service as is. It will probably continue to fluctuate widely in performance. But now, if a client connects to the API using HTTPS and provides contact information either in their User-Agent header or as a parameter on their queries, then we will send them to a separate pool of machines. We expect to be able to better control the performance of these machines because, if a script starts causing problems, we can contact the people responsible for the script to ask them to fix it. Or, in extremis, we can block it.
How does it work? Simple. You can do one of two things to get directed to the "polite pool":
- Include a "mailto" parameter in your query. For example: https://api.crossref.org/works?filter=has-full-text:true&mailto=GroovyBib@example.org
- Include a "mailto:" in your User-Agent header. For example: GroovyBib/1.1 (https://example.org/GroovyBib/; mailto:GroovyBib@example.org) BasedOnFunkyLib/1.4.
Note that this only works if you query the API using HTTPS. You really should be doing that anyway (wags finger).
So I don't want to limit people from using the API, but I also want to ensure everyone's playing nicely. Maybe a way to solve this is to enforce that an email address is included in the URL. That might be enough, but an additional idea might be to send a one-time email to that address with a link to confirm that it's valid and belongs to the API caller.
This shouldn't limit the queries being made by reasonable users, but will let me get in touch with users who are, ahem, overusing the API in a way that limits my ability to maintain it for free.