DedSecInside / gotor

This program provides efficient web scraping services for Tor and non-Tor sites. The program has both a CLI and REST API.
GNU General Public License v3.0
159 stars 44 forks source link

Handle 429 status code with retry #43

Open KingAkeem opened 1 year ago

KingAkeem commented 1 year ago

429 indicates too many requests and possibly has a header attached that indicates when another request should be retried Retry-After.

This could be used to pull those requests into a separate "retry" channel that will retry those requests after the Retry-After period has passed or a default time period. This would help to eliminate false negatives for sites that are actually up and running.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429 https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Retry-After