search

gt-big-data / QDoc

Quick & Dirty Operating Crawler
4 stars 1 forks source link

There are multiple feeds for thehour.com #31

Open simon0929 opened 8 years ago

simon0929 commented 8 years ago 
> use big_data
switched to db big_data
> db.feed.find({'feed': /thehour.com/})
{ "_id" : ObjectId("56af7b52926cb76b598c1156"), "feed" : "http://www.thehour.com/search/?f=rss&t=article&c=stamford_times&l=50&s=start_time&sd=desc", "active" : true, "lastCrawl" : ISODate("2016-02-22T17:48:04.507Z"), "crawlFreq" : 3600, "stamp" : ISODate("2016-02-20T22:19:00Z") }
{ "_id" : ObjectId("56af7b52926cb76b598c1394"), "feed" : "http://www.thehour.com/mobile/search/?f=rss&t=article&c=wilton_villager&l=50&s=start_time&sd=desc", "active" : false, "lastCrawl" : ISODate("2016-02-01T15:31:59Z"), "crawlFreq" : 240 }
{ "_id" : ObjectId("56c75364926cb76b598da0ca"), "feed" : "http://www.thehour.com/search/?f=rss&t=article&c=stamford_times&l=50&s=start_time&sd=desc", "stamp" : ISODate("2016-02-16T22:05:00Z"), "lastCrawl" : ISODate("2016-02-19T16:52:25.630Z") }
{ "_id" : ObjectId("56c75385926cb76b598da0d2"), "feed" : "http://www.thehour.com/search/?f=rss&t=article&c=stamford_times&l=50&s=start_time&sd=desc", "stamp" : ISODate("2016-02-16T22:05:00Z"), "lastCrawl" : ISODate("2016-02-19T16:54:35.214Z") }
{ "_id" : ObjectId("56c75a72926cb76b598da452"), "feed" : "http://www.thehour.com/search/?f=rss&t=article&c=stamford_times&l=50&s=start_time&sd=desc", "stamp" : ISODate("2016-02-16T22:05:00Z"), "lastCrawl" : ISODate("2016-02-19T17:28:36.304Z") }