search

alephdata / memorious

Lightweight web scraping toolkit for documents and structured data.
https://docs.alephdata.org/developers/memorious
MIT License
311 stars 59 forks source link

Introduce sampling_rate to run a subset of crawler tasks in debug mode #129

Closed sunu closed 4 years ago

sunu commented 4 years ago

refs #56

This is not as easy to use as a --sample cli flag as mentioned in #56 but was the easier choice implementation wise and a good middle ground imo.