Stop the scrape if too many 404

openzim / wikihow

WikiHow scraper

https://download.kiwix.org/zim/wikihow/

GNU General Public License v3.0

15 stars 2 forks source link

Closed kelson42 closed 2 years ago

kelson42 commented 2 years ago

We probably should make a threshold based on a percentage of failure over the total number of articles.

rgaudin commented 2 years ago

What percentage should it be? Should it be configurable?

kelson42 commented 2 years ago

Configurable, yes. 0% per default.