search

istresearch / scrapy-cluster

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
http://scrapy-cluster.readthedocs.io/
MIT License
1.18k stars 323 forks source link

Adding support of shared cookies for requests having the same crawlid #208

Closed MohamedMb closed 5 years ago

MohamedMb commented 5 years ago

Adding a custom download middleware distributed_cookies.DistributedCookiesMiddleware that use Redis to set and get the serialized cookie. Adding a custom spider middleware distributed_cookies.ClearCookiesMiddleware that delete cookie if an item with the same crawlid is yield. It's also possible to set a cookie with automatic expiration (time in ms in settings).

MohamedMb commented 5 years ago

Do not take it into account. See #208