Adaptive fetch schedule for feeds

[sebastian-nagel]

At present, the refetch schedule for seed feeds is globally 3 hours which is a compromise between

• trying not to miss links for large national newspapers (10 min. are required to get every news link in rush hours, e.g., Monday morning)
• not overloading small regional newspapers or news blogs with only a few news per day
  • or dedicated per-category feeds (eg "animals") or reginal / per-town feeds (eg "Mezzogiorno Caserta")
  • or stale and empty feeds

The schedule should adapt to the change frequency within a configurable min and max refetch interval (eg., 10 min. - 2 weeks). Detection of unchanged feeds should be independent of a last-modified time sent together with the server response.

[sebastian-nagel]

Adaptive scheduler in production since 3 weeks:

• the mentioned infrequently changing RSS feeds now have a long fetch interval: 1 week for animals from Corriere della Sera
• frequently changing feeds (e.g., Guardian politics) are re-fetched within the configured minimum interval

[sebastian-nagel]

Testing in production ongoing, waiting for DigitalPebble/storm-crawler#418