philippta / flyscrape

Flyscrape is a command-line web scraping tool designed for those without advanced programming skills.
https://flyscrape.com
Mozilla Public License 2.0
1.02k stars 29 forks source link

Retry #34

Closed philippta closed 7 months ago

philippta commented 7 months ago

A new module should be added that retries failed requests. A request is considered failed if either the connection could not be established or if the response status code indicates a failure.

Status codes that should be retried:

A Retry-After header should be honored if present in the response. When no Retry-After header is provided, requests should be retried 3 times with an exponential back-off.

This module should come before the ratelimit module in the call-chain so that the internal rate-limiting is honored.