How does feedland decide which feed to check next?

[cagrimmett]

This is not time sensitive, but Dave suggested that has the potential for big performance improvement in FeedLand.

it's very simple right now, and i think that the news flow suffers for its simplicity.

How do other products do it? What kind of algorithm do we want powering this?

[scripting]

Thanks for starting this thread Chuck.

Here are two feeds that are treated the same by the selection algorithm.

1. Feed X is subscribed to by 1 person who used FeedLand once three months ago and loaded in his OPML file and one of the feeds is subscribed to a very active Telegram channel, and when we check it we get 3500 new items for the database.

2. Matt's feed is subscribed to by everyone at Automattic, but by some quirk his feed doesn't support rssCloud. He writes five posts a day, on average we find out about them an hour after they're posted.

The point is that these two feeds deserve very different treatment. Should we be reading Feed X at all until the one subscriber shows up again?

The fact that a lot of active people subscribe to Matt and he updates regularly says he should get a higher priority.

[scripting]

I think there would be two algorithms each running in their own thread.

The fast thread and the slow thread. Say 10 percent of the feeds are in the first and the rest are in the slow thread.

They run at the same rate, which means the fast feeds get checked 10 times for every time the slow ones get checked.

This structure is easy to implement.