Skip links-extracted when reached limit for domain

scrapinghub / frontera

A scalable frontier for web crawlers

BSD 3-Clause "New" or "Revised" License

1.29k stars 216 forks source link

Closed sibiryakov closed 6 years ago

sibiryakov commented 6 years ago

Write stats about dropped LE events Add filter_extracted_links method to API