scripting
commented
6 years ago My SQL confidence is getting up there. I was able to modify the query for the hotlist to exclude items with null titles, as follows --
const sqltext = "SELECT subscriptions.feedUrl, feeds.title, feeds.htmlUrl, COUNT(subscriptions.feedUrl) AS countSubs FROM subscriptions, feeds WHERE subscriptions.feedUrl = feeds.feedUrl and feeds.title is not null GROUP BY feedUrl ORDER BY countSubs DESC LIMIT 100;";
I'm going very slowly and carefully through the database routines, and testing stuff, and collecting my thoughts and making decisions.
For example, what to do when there's an error reading a feed?
If it's a feed that used to work, the answer is clear, continue to read it, and increment ctErrors and ctConsecutiveErrors. Leave it in the database because a row in the database is cheap.
Then, what to do if there's an error the first time you read a feed? Now we don't have a title, or even an htmlUrl to associate with the feed. At first I thought -- don't add it to the database. But then I changed my mind. The user could try and try again to subscribe to the feed, without much effort, because it's in their OPML file. I have many such feeds in my oldest subscription list.
I am happy to keep checking it, but only on a regular basis, not "on demand."
So it also goes in the database, in the feeds table.
Now I have to make the hotlist aware of these elements in the table.
I want to exclude feeds from the hotlist if they have a NULL title or more than 5 consecutive errors. That would mean we checked the feed over a period of hours and it has continued to be down. We will keep checking, and if it reappears it will also reappear in the hotlist.