Handle 404 for article snapshot update

disinfoRG / ZeroScraper

Web scraper made by 0archive.

https://0archive.tw

MIT License

10 stars 2 forks source link

Closed pm5 closed 4 years ago

pm5 commented 4 years ago

Stop taking snapshots if hit 404 by setting next_snapshot_at to 0 as in other cases. Also made spiders and db pipeline better separated.

pm5 commented 4 years ago

A few things added besides fixing #69:

1d2d0b76811b1e41ae4cea2b306ebee6ccdc611d so that failing to insert snapshot will rollback updates on the article.
changes up to 82c6f554eab8eb26826e0db21d8e6a3eb0da4712 so that pipeline does not depend on spider.name any more, making it feasible to add more spiders so long as they allow the protocol.