Jitter

[philippta]

A new module should be added that introduces a jitter effect, delaying or advancing the rate at which requests go out.

For now the jitter time should be fixed at 10% of the request rate:

• 1 requests / second => 1000ms in-between requests ± 100ms jitter
• 10 requests / second => 100ms in-between requests ± 10ms jitter
• 100 requests / second => 10ms in-between requests ± 1ms jitter

[philippta]

After thinking about this change a little more it doesn't seem to add much value.

The original idea behind this was to make the scraper appear more "human" by adding random delays to the request. However, it would be a far fetch to assume this would make it more human-alike.

Technically modern services that have ratelimiting do not care much about the request pattern. They just calculate the number of request in a given time frame.

For more info Cloudflare has a public post how they do ratelimiting: https://blog.cloudflare.com/counting-things-a-lot-of-different-things