alephdata / memorious

Lightweight web scraping toolkit for documents and structured data.
https://docs.alephdata.org/developers/memorious
MIT License
311 stars 59 forks source link

Memorious session information expiration #203

Open monneyboi opened 2 years ago

monneyboi commented 2 years ago

I'm running a crawler that does a lot of recursion, so much that at some point, the stored session information in Redis expires, and the scraper stops using the configured proxy.

This could be solved by increasing the expiration time: https://github.com/alephdata/memorious/blob/da7291aa34517fd1efba3ab47e8d752a40510693/memorious/logic/http.py#L98

sunu commented 2 years ago

In that case, we can make REDIS_SHORT configurable in servicelayer: https://github.com/alephdata/servicelayer/blob/d1bac4c8f1991f07a2df2720099ea4e7466228c7/servicelayer/settings.py#L7