search

alephdata / memorious

Lightweight web scraping toolkit for documents and structured data.
https://docs.alephdata.org/developers/memorious
MIT License
311 stars 59 forks source link

Option to run a multithreaded worker in sync mode. #169

Closed sunu closed 3 years ago

sunu commented 3 years ago

Multithreaded worker doesn't work with fakeredis or sqlite. So a Redis server URI has to be set in REDIS_URL and a database server URI has to be set in MEMORIOUS_DATASTORE_URI.

Calling syntax looks like: memorious sync crawler_name --threads 4