Open monneyboi opened 2 years ago
I'm running a crawler that does a lot of recursion, so much that at some point, the stored session information in Redis expires, and the scraper stops using the configured proxy.
This could be solved by increasing the expiration time: https://github.com/alephdata/memorious/blob/da7291aa34517fd1efba3ab47e8d752a40510693/memorious/logic/http.py#L98
In that case, we can make REDIS_SHORT configurable in servicelayer: https://github.com/alephdata/servicelayer/blob/d1bac4c8f1991f07a2df2720099ea4e7466228c7/servicelayer/settings.py#L7
REDIS_SHORT
I'm running a crawler that does a lot of recursion, so much that at some point, the stored session information in Redis expires, and the scraper stops using the configured proxy.
This could be solved by increasing the expiration time: https://github.com/alephdata/memorious/blob/da7291aa34517fd1efba3ab47e8d752a40510693/memorious/logic/http.py#L98