search

mediacloud / rss-fetcher

Intelligently fetch lists of URLs from a large collection of RSS Feeds as part of the Media Cloud Directory.
https://search.mediacloud.org/directory
Apache License 2.0
5 stars 5 forks source link

Implement undead/zombie feeds? #38

Open philbudne opened 4 months ago

philbudne commented 4 months ago

@NullPxl suggested that instead of auto-disabling feeds, we instead poll them very slowly.

One concern is how to signal this status to users looking at the web search "catalog" of sources & feeds.

The crudest way to do this would be to continue to set system_enabled to false, but to only use it to control the upper limit of poll interval (that is, select an alternate MAXIMUM_BACKOFF_MINS value, perhaps MAXIMUM_ZOMBIE_BACKOFF_DAYS?). The UI could just display zombie or poop emoji instead of a red X.

However, system_enabled is used in the staging database to limit the number of feeds that are actually polled. Adding a system_undead column boolean column, or making system_enabled a tristate would allow explicit disabling of feeds.

NullPxl commented 4 months ago

I like the tristate idea, is this how you would imagine system_enabled working? 0 (❌) - Explicitly set to disable the feed. Used in staging database to limit polls as you mentioned. 1 (✅) - The feed is working (or has not yet hit the failure threshold). 2 (💀/💩/🧟) - Feed failure threshold hit (such as the current points system); polling is significantly reduced.

Adding a little tooltip on hover for the zombie/skull feeds to say something like Unable to reach feed - polling has been reduced might be helpful. (or a legend somewhere)