Playlist updates with short max backlog

[wavexx]

Describe the bug I noticed some subscriptions I had didn't receive any update. Could it be that playlists are downloaded/processed in yt order, truncating results to max_update_backlog?

If so, a playlist that has new videos appended in chronological order will never receive any update.

To Reproduce This playlist is in ascending chronological order: https://www.youtube.com/playlist?list=PLm9GPtnKLH_xb_griZ4Qg4eXLFLeLEO5d

ytcct subscribe test [url]

only shows the oldest videos being processed.

Expected behavior ytcc shouldn't be influenced by the playlist order. I'm not sure if there's a way to force time sorting, but if not some flag to indicate that the playlist is reversed would be needed?

[woefe]

Could it be that playlists are downloaded/processed in yt order, truncating results to max_update_backlog?

Yes, exactly that is the case. There is no real solution to this issue due to how youtube_dl fetches playlists. As a workaround either increase max_update_backlog to a very large number, which will probably increase the time required to update your subscriptions. Or in this particular case subscribe https://www.youtube.com/c/FredrikKnudsen/videos

[wavexx]

I guess there's no way to fetch the list reversed through a parameter?

[woefe]

AFAIK, there is no URL parameter on YouTube. Just read the youtube_dl manpage again and found --playlist-reverse. Not sure how it reverses the list, however. If it does not have to check the entire list to reverse it, it might be a better option than increasing max_update_backlog. I'll have a look (probably next week).

[wavexx]

On Sun, Feb 14 2021, Wolfgang Popp wrote:

found --playlist-reverse

That's what I was thinking when I reported this.

If it turns out that this is done internally in ytdl, I'd still wish that there was a flag to indicate that the playlist is appended to, and have ytcc handle that by fetching the full playlist, reversing it and then truncate it according to the max backlog.

The reason is that increasing the backlog for all lists slows down the update, and also generates warnings about the age limit, making that useless too.

I have a handful of cases where I can't find an alternate playlist with videos on the correct order, since the main channel includes other stuff I'm not interested in subscribing.

[woefe]

I've looked into --playlist-reverse. It does not work together with pagination as I wished. It does not reverse the entire playlist, but only reverses what is retrieved until max_backlog.

I figured out a solution and implemented #88. @wavexx can you test, if that solves your use-case? Basically, there is now a --reverse flag on the subscribe command. To reverse the playlist ytcc has to load all playlist entries first, then reverse it and finally slice it to max_backlog length. Loading the entire playlist makes updating quite slow for large playlists. Unfortunately, I could not find a better way, and standalone youtube-dl has to do the same.

[woefe]

@wavexx make sure to have a backup of your database when you test it. There is a database migration happening on the first start, and there are currently no rollbacks of those migrations.

[wavexx]

Would it make sense to automatically detect if the playlist is reversed during "subscribe"? Looks trivial to do, and might be a nice ergonomic improvement as well, making subscribe just-work. A warning or notice when this happens would also be nice to inform the user that updates will not be fast.

[woefe]

How would you actually detect if a playlist is reversed?

[wavexx]

I didn't check the format.. Isn't there a date of the video in there without having to fetch the video itself? :/

[woefe]

Checking by publish date is not reliable because

• Videos don't have to be in publish date order (or reverse).
• Publish date is not always available
• Videos published on the same day have the same publish date (see #78)

I find unreliable autodetect annoying. That's why I want to let the user decide whether or not a playlist is being appended or prepended to. Some kind of hint could still be useful, though.

[wavexx]

Agree. I like the idea about the hint though, would remove a source of potential surprises.

[woefe]

What's implemented now in #88 is following behavior:

• optional --reverse flag on subscribe command. When given, the playlist is checked in reverse order
• warning, if playlist seems to be in wrong order when subscribing
• reverse command to change the update behavior of a playlist

[wavexx]

I tested this for the last two days and everything went well. Upgrade worked just fine. I was able to reverse old subscriptions too.

Ironically I couldn't find a way to change the URL of an existing subscriptions when I wanted to use an alternate url (may subscribe --force could be coerced to do that?).