Database issues

[scripting]

1. On both systems I'm subscribed to the Art Feeds reading list.

2. I added an Art category to my tabbed news page. On feedland.org, I get what I expect, pictures from the feeds in the Art category. On feedland.com I get an error dialog: Can't get the river because there are no feeds in the "Art" category

3. When I go to the Feed List page for the Art category on both systems, the feeds of the Art Feeds reading list show, in green.

The next step in getting to the bottom of this is to look at the code behind the feed list and behind river generation to see how they might ask about categories.

[scripting]

Good news! I have been able to reproduce the problem on feedland.org, which is great because:

1. The problem is with my code, no compatibility issues in MySQL versions.
2. I can fully debug on feedland.org, i can do command-line MySQL queries, and step through my server code in the debugger.
3. It was immediately obvious what the problem was once I did a command-line query.
4. The (new) reading-list code is implicated.

This is one of the fundamental laws of programming. A problem that seems insurmountable often succumbs to a good night's sleep.

[scripting]

Here's the mistake, as viewed through a slice of my subscriptions.

You can see one of the subscriptions comes from a reading list.

And you can also see the convention of how categories are stored, embedded in commas and separated by commas.

If they're formatted incorrectly, and they are for reading list-sourced subscriptions, the River algorithm won't find them.

If all the subscriptions came from the art reading list, then they'll never be found and thus never show up in a river, which is 1/2 of the observed bad behavior.

So why is it different for Feed Lists? Because that selection is done on the client, not the server, based on selecting feeds from the user's subscription list, which is always present in the client. Its version of the feed record has the categories parsed into an array, which is the high level way of doing this (as opposed to storage, which must be efficient and obey the rules of SQL). So it worked there.

Still thinking about how to address this...

[scripting]

This is where the error is, in feedlanddatabase:

function getCategoriesFor (feedUrl) { //10/25/23 by DW
    var categories = undefined;
    theNodeArray.forEach (function (item) {
        if (item.xmlUrl == feedUrl) {
            categories = item.category;
            }
        });
    return (categories);
    }

The correction is in the last statement, it should be:

return ("," + categories + ",");

[scripting]

The problem is fixed. ;-)

New versions of feedland and feedlanddatabase, 0.6.26 and 0.7.16.

Notes for users.

[cagrimmett]

@scripting I think this may still be an issue to some degree. I subscribed to the RSS Club reading list earlier today, yet when I went back tonight to look at it, I checked the feed list for the RSS Club reading list and noticed that only a handful were checked (presumably ones I had subscribed to already). I needed to unsubscribe and resubscribe before the feeds were checked for me.

Since FeedLand.com is running the latest version and I already unsubscribed and resubscribed to my other reading lists last week when you fixed this problem, I am concerned it is still an issue. I don't have other reading lists handy to test, but I will find some and report back.

[scripting]

@cagrimmett — I saw this too when I subscribed to the list yesterday.

[scripting]

@cagrimmett -- this is the big one for today. luckily the same problem appears on feedland.org where the debugging tools are orders of magnitude better. Here's a screen shot of the feed list page for the reading list on feedland.org.

[scripting]

When I tried clicking on the title of the first feed, I was surprised to hear it wasn't in the database.

It is subscribed to, this is so weird, but once again -- the categories are kind of wrong. ;-)

[scripting]

Then I asked directly if there is a feed record.

select title from feeds where feedurl = 'http://www.aaronblakeley.com/feed/';

Nothing came back. In fact it is not in the feed list.

[scripting]

So what's unusual about this reading list?

https://daverupert.com/rss-club/feeds.xml

No category elements. That accounts probably for ,undefined, as the value of the categories column for each.

I haven't looked at the code yet, but this is where I'm going to start. ;-)

[scripting]

@cagrimmett -- question, when you unsubbed from the RSS Club list, did you say you resubbed?

I expect you didn't, but if you did, the incorrect behavior would repeat.

Verify?

[cagrimmett]

@scripting Yes, I unsubbed and resubbed to the RSS Club list last night, and since resubbing, it looks like I am subscribed to all of the feeds on FeedLand.com:

I have not tried to subscribe on FeedLand.org, but if it would be helpful for me to do that I can.

[scripting]

@cagrimmett -- I will have to ponder this! :-)

[scripting]

In any case, I'm about to wipe the bug off the face of the earth permanently and unilaterally. No more Mr Nice Guy.

[scripting]

Here's what I found out.

1. The unchecked items really weren't in the feeds table in the database. So there was nothing wrong with the lack of a checkmark.
2. The way reading lists work, the checking of feeds existence in the feeds table wouldn't happen until the reading list was checked, in its rotation. And the code that did the checking for the feedrec did not emit specific error messages for the inability to create a feedrec, just "hey there was a problem doing this."
3. I decided it shouldn't wait -- this should be part of the process of adding a reading list, not just checking one. so any errors that happened would happen on creation, and I added console.log messages that tell the story. on both success and failure.
4. there were a small number of feeds in that list that really do have errors. why there were so many yesterday, i don't know, can't reproduce now, i'm getting mostly checkboxes.
5. reading lists will have bad feeds in them. but we should not pass on the bad feeds in our OPML lists, in other words FeedLand should be a place that cleans these things up. just have to make sure not to output any bad feed URLs in the OPML files we generate.
6. A new version is coming shortly.

[scripting]

BTW -- good thing you reported this @cagrimmett -- this would have turned up as a problem at some point, and i'd much rather do this kind of work without the pressure of users waiting for it, and us looking bad. i have to overcome a rep for producing systems that need users to find the most basic errors, but i had no choice up till now. now with you all on board, we do this like professionals. so you know -- i reallllly appreciate the help.

[cagrimmett]

@scripting New wrinkle: I subscribed to a new reading list tonight (https://feedland.com/?readinglist=https%3A%2F%2Fdefaults.rknight.me%2Fopml.xml) and encountered something strange.

The list is supposed to have 206 feeds, but after I subscribed I went to view the feed list and it is only showing 6 feeds, all of which I think is only feeds that I had previously subscribed to and are in this particular reading list.

[scripting]

I didn't subscribe to the list, but went to the feed list from the reading list page, and got a slightly different screen shot.

It's possible it wasn't able to read all the other files. If it wasn't able to read them, they don't show up on the feed list page.

It does seem like a lot of feeds to be bad, but that's the next thing to check. Was FeedLand wrong??

[scripting]

I picked one feed at random and tried running into through the Feeder app.

http://feeder.scripting.com/returnjson?feedurl=https://yuexun.me/rss.xml

After a programmer error, which I corrected, it works.

This is the whole OPML file.

[scripting]

Just tried subscribing to the list on feedland.org, a lot more feeds came back.

https://feedland.org/?subscriptionlist=https%3A%2F%2Fdefaults.rknight.me%2Fopml.xml

BTW, I have no idea what this means. ;-)