Sometimes the API gives an HTTP 429 Too Many Requests error even with the rate limiters in place. On receiving such an error, it would probably be best to take some kind of action for the crawler as a whole and delay. Otherwise, the IP address might get temporarily locked out.
Sometimes the API gives an HTTP 429 Too Many Requests error even with the rate limiters in place. On receiving such an error, it would probably be best to take some kind of action for the crawler as a whole and delay. Otherwise, the IP address might get temporarily locked out.