search

scrapinghub / frontera

A scalable frontier for web crawlers
BSD 3-Clause "New" or "Revised" License
1.29k stars 216 forks source link

more precise scoring, encoding from scrapy, hash for request #316

Closed sibiryakov closed 6 years ago