Worker should follow instructions in HTTP 429 responses

dice-group / Squirrel

Squirrel searches and collects Linked Data

Other

23 stars 19 forks source link

Worker should follow instructions in HTTP 429 responses #121

Open MichaelRoeder opened 5 years ago

MichaelRoeder commented 5 years ago

Description

As a website owner, I expect crawlers to behave. If they are crawling too fast and my server responds with HTTP 429 (RFC 6585), the crawler should follow the instructions and slow down.

MichaelRoeder commented 4 years ago

Note: If Web API returns status code 429, it means that you have sent too many requests. When this happens, check the Retry-After header, where you will see a number displayed. This is the number of seconds that you need to wait, before you try your request again.