How to stop if encountered visited links?

scrapy-plugins / scrapy-deltafetch

Scrapy spider middleware to ignore requests to pages containing items seen in previous crawls

267 stars 48 forks source link

Open mosynaq opened 6 years ago

mosynaq commented 6 years ago

Hi! Is it possible to set deltafetch stop scrapy crawling when encountering a visited link? I really need this!

ghost commented 6 years ago

I was asking myself the same question, I think you can do it simply by passing the meta parameter deltafetch_key to your spider, and defining the url as your key. https://blog.scrapinghub.com/2016/07/20/scrapy-tips-from-the-pros-july-2016