TeamHG-Memex / scrapy-crawl-once

Scrapy middleware which allows to crawl only new content
MIT License
79 stars 23 forks source link