How to handle stale data

ngshiheng / burplist

Web crawler for Burplist, a search engine for craft beers in Singapore

https://burplist.com

MIT License

11 stars 5 forks source link

How to handle stale data #37

Closed ngshiheng closed 3 years ago

ngshiheng commented 3 years ago

i.e. cases where the URL is no longer valid (product deleted)
We simply can't just determine a product is no longer valid just because the URL changes
What if the URL changes but the same product still remains with new URL?

ngshiheng commented 3 years ago

product model is updated with updated_on date. We can use that as last_check date and filter out all the products where last_check is > X days/weeks. Done.