Automatically increase politeness delay if received 420 or 429 HTTP code

jesbin / crawler4j

Automatically exported from code.google.com/p/crawler4j

0 stars 0 forks source link

Open GoogleCodeExporter opened 8 years ago

GoogleCodeExporter commented 8 years ago

It would be super amazing if crawler4j could slow request speed if receive and 
HTTP 420 or HTTP 429

Original issue reported on code.google.com by neilmcgu...@gmail.com on 22 Nov 2013 at 2:04

GoogleCodeExporter commented 8 years ago

Original comment by avrah...@gmail.com on 18 Aug 2014 at 3:46