search

maddyblue / goread

RSS reader in go on app engine; formerly goread.io
ISC License
2.38k stars 295 forks source link

check interval is too long for some feeds #314

Open mblakele opened 9 years ago

mblakele commented 9 years ago

I added http://www.economist.com/blogs/freeexchange/index.xml to my feeds and marked all read. Then I noticed another RSS reader was reporting a new article, but goread wasn't. Looking at the feed details, it says it was last updated an hour ago and won't update again for seven hours.

If the goal is to avoid excess polling, it might help to consider the times of day when sites generally update. This particular blog tends to update overnight and in the morning hours — relative to Pacific time. It might make sense to look at this history, check relatively frequently during those hours, then back off during hours when the feed is usually idle.

mblakele commented 9 years ago

Another use case to consider is a low-traffic stackoverflow feed like http://stackoverflow.com/feeds/tag?tagnames=elisp&sort=newest. It gets updates relatively infrequently, but users may want to see new items relatively quickly. This might be handled by specifying site-specific max check intervals as a configuration option.

maddyblue commented 9 years ago

goread supports pubsubhubbub, a protocol which, if supported by the publisher, will push updates to goread instantly when new articles are published. For sites that don't support that, though, there's currently an 8 hour maximum polling time, and down to 20 minutes for feeds that generally update more. If this is a new feed then it make take some time for the expected update interval thing to configure itself and the polling time to go down.

This is a hard problem, and I agree it'd be best if goread could predict the normal updates times, since they might not happen during the night. I am unlikely to do that work soon, however.