Add a list of past crawl times to feed. - Githubissues

gt-big-data / QDoc

Quick & Dirty Operating Crawler

4 stars 1 forks source link

Add a list of past crawl times to feed. #34

Open supersam654 opened 8 years ago

supersam654 commented 8 years ago

It would be really nice to know when we last crawled feeds.

Requirements:

Add an array called crawlTimes to each feed.
Whenever we crawl a feed, push the current time into the front of the crawlTimes array for that feed.
Remove the last element from the array if there are more than 50 times in the array.

See https://docs.mongodb.org/manual/tutorial/limit-number-of-elements-in-updated-array/ for a way to get Mongo to shrink the array when it gets too big.