Adding support of shared cookies for requests having the same crawlid

istresearch / scrapy-cluster

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

http://scrapy-cluster.readthedocs.io/

MIT License

1.18k stars 323 forks source link

Adding support of shared cookies for requests having the same crawlid #209

Open MohamedMb opened 5 years ago

MohamedMb commented 5 years ago

Adding a custom download middleware distributed_cookies.DistributedCookiesMiddleware that uses Redis to set and get the serialized cookie. Adding a custom spider middleware distributed_cookies.ClearCookiesMiddleware that delete cookie if an item with the same crawlid is yield. It's also possible to set a cookie with automatic expiration (time in ms in settings).

coveralls commented 5 years ago

Coverage decreased (-1.7%) to 69.212% when pulling 65602317475076dfdc99a5bd9141a762e62a4cbe on MohamedMb:dev into 55b784d1220927ef628be9b1cb0452b1cb4e247c on istresearch:dev.

coveralls commented 5 years ago

Coverage decreased (-1.7%) to 69.212% when pulling 65602317475076dfdc99a5bd9141a762e62a4cbe on MohamedMb:dev into 55b784d1220927ef628be9b1cb0452b1cb4e247c on istresearch:dev.

coveralls commented 5 years ago

Coverage decreased (-0.2%) to 70.804% when pulling e42fb8f049cecbc27122ff01beda9deb94c30cda on MohamedMb:dev into 55b784d1220927ef628be9b1cb0452b1cb4e247c on istresearch:dev.