Our current workflow for the crawler produces no good_nodes on its own. This PR fixes this problem by introducing a rate limiter, only allowing 25 requests per second.
Changes:
Adds a new dependency, governor.
Introduces the Limiter struct, a wrapper around governor::RateLimiter along with governor::Jitter (to prevent thundering herd problem).
Awaits Limiter::until_ready before sending crawl requests, to not need send more than MAX_REQUESTS_PER_SEC requests per second.
Our current workflow for the crawler produces no
good_nodes
on its own. This PR fixes this problem by introducing a rate limiter, only allowing25
requests per second.Changes:
governor
.Limiter
struct, a wrapper aroundgovernor::RateLimiter
along withgovernor::Jitter
(to prevent thundering herd problem).Limiter::until_ready
before sending crawl requests, to not need send more thanMAX_REQUESTS_PER_SEC
requests per second.