Make portable playlists

[icymist]

One problem that I face when making playlists on YouTube is that after spending time to make a playlist, sometimes the videos disappear (video made private, video deleted, etc.) without leaving behind any information about the deleted item.

I think this can be overcome by MPS by doing the following:

1. Add config options to save playlists to a desired location
2. Make playlists portable by writing them in yaml with enough information about the item, so that in case the original video is deleted, one at least has enough information about it to add a new source url. Also, syncing across computer will become feasible.

[ids1024]

I would also like to see mpsyt use a standard text-based playlist. Until then, perhaps this script would be useful. The first column is the length of the song in seconds.

#!/usr/bin/python

import sys
import pickle
import os

path = os.path.expanduser('~/.config/mps-youtube/playlist_v2')
with open(path, 'rb') as file:
    playlists = pickle.load(file)

for name, playlist in playlists.items():
    print('\n'+name + '\n' + len(name)*'-')
    for song in playlist.songs:
        print(song.length, song.ytid, song.title, sep='\t')

[ids1024]

@np1 Do you have anything against storing playlists and configuration in a json based format? Plain text formats are more user friendly and I personally find no benefit to using pickle.

[np1]

No I have nothing against that, I think it would be good :smile:

[Atlantic777]

Ok, let's bump this one. Here are my thoughts:

• Items should have ytid, URL or something so we can quickly load them without querying yt API
• In case link is broken (in example, the video is deleted) we should have enough information to search for alternative. A song matcher is already implemented in album search, so we can rely on that one. It uses song duration and title. There's an open ended question how to extract song title from the video title. I think that video title isn't good enough to use it as is, but we can strip it down using some common words (video, vevo, album, live, rare...)
• A nice feature would be to check playlist integrity (check if all items are available) and search for substitutions (manually or automatically)
• Manual matching if match confidence is low. Let user search for item and pick desired or let him skip that item if alternative is really not available.

I find writing JSON files by hand a bit difficult. I prefer YAML when there is need for users to build such files manually. In that case, our playlist could be implemented in such way that it can work on playlist with only some information available. In example, only song title is required, and playlist loader should expand and add other fields upon loading the file, so a user can type in the rest or fill it interactively (either manually or automatically).

Also, I implemented a basic playlist reconstruction from text files in #361. I would still like to support that way and I see a few solutions.

1. those two features implemented separately (I don't like this one)
2. add header, like #mpsyt-raw and #mpsyt-yaml, and trigger needed parser based on that
3. make parser guess which format is used (easier for users but a bit more complex code and I think that it's not worth it)
4. make some sort of convert script (built separate tool, I don't like this either)
5. insist to use only YAML style playlists

[ids1024]

@Atlantic777 Yaml is fine, but it would add another dependency. Not that I think that is a major issue, but it is worth mentioning.

You can see what is currently stored in a playlist in https://github.com/mps-youtube/mps-youtube/blob/develop/mps_youtube/playlist.py. For each video, there is the id, title, and length (in seconds).

I think it would be best to use yaml (or json, as I already suggested) instead of the current binary format, though there should be code to convert the playlist someone already has. There is not really a need to have two formats.

[ids1024]

We could also use m3u format, which is a standard for playlists. Note that the youtube urls will not work in most software (though it does in mpv), but media urls would not really be appropriate. The user can just create a file with a list of urls, one on each line, and the metadata could be added by mpsyt.

#EXTM3U

#EXTINF:420, Rush-"The Garden"-2012 HQ
https://youtube.com/watch?v=1wztuqskio8
#EXTINF:308, Rush: Nobody's Hero
https://youtube.com/watch?v=Axbcg9obGf4
#EXTINF:335, Rush - Subdivisions (with lyrics)
https://youtube.com/watch?v=Vf8jvSPA3XQ

The one thing this does not handle is playlist title, but that could be encoded in the filename, or a special comment.

Of course, we could use a text based format of own own. Just requiring the user to put an url on each line and retrieving the rest seems good, and would be desired if we were to create our own format.

[Atlantic777]

I don't see any obvious benefits of using m3u. Maybe I'm missing something. I think that YAML/JSON custom format leaves more space for improvements and doesn't add much overhead. Also, if we find that there's need for m3u or raw list of youtube links, we could add simple wrappers which would convert to and from that formats and our custom text format.

By the way, I feel a need to share my opinion why I prefer YAML over JSON. I find it a bit daunting to debug opening and closing curly brackets in JS-like files.

[ids1024]

There is not much of an advantage to m3u, but it too could be extended. The #EXTINF are from ext33u (extendend m3u) which is backwards compatable with the basic format, which is just a list of urls. We could add a #MPSYTINF or similar to add our own metadata.

An advantage to m3u is that we could implement a simple parser for it, while yaml would pull in another hard dependency. It also would be very user friendly to just write a list of urls to manually create a playlist.

So, yaml would looks something like this:

title: A playlist
items:
    - url: "https://youtube.com/watch?v=1wztuqskio8"
      title: Rush-"The Garden"-2012 HQ
      length: 420

    - url: "https://youtube.com/watch?v=Axbcg9obGf4"
      title: "Rush: Nobody's Hero"
      length: 308

    - url: "https://youtube.com/watch?v=Vf8jvSPA3XQ"
      title: Rush - Subdivisions (with lyrics)
      length: 335

(Some strings have quotes to escape the colons in them)

That certainly looks harder to write than just a file with an url on each line, though of course the user would not need to provide the title and length lines. Or do you know a better way to format it?

[Atlantic777]

Well, what about this?

# required
title : my playlist

# required 
song_urls : [
"url0",
"url1",
"url2",
"url3",
"url4",
"url5",
]

# generated on first load
meta:
  - id     : 0             # references songs url array
    title  : Song title    # doesn't have to match video title
    length : 200           # just for matching purposes

  - id     : 1
    title  : Song title 2
    length : 205

Required part is easy to type in, and meta data part is autogenerated. If we go that way, JSON would look similar except some brackets more. Enclosing urls in quotes is required for both JSON and YAML which can be a significant overhead for users in comparison to the m3u way. I'm not sure if those #EXTINF tags are required in m3u. Are they?

Here's the JSON format for sake of comparison.

{
  "title": "my playlist", 
  "song_urls": [
    "url0", 
    "url1", 
    "url2", 
    "url3", 
    "url4", 
    "url5"
  ],
 "meta": [
    {
      "length": 200, 
      "id": 0, 
      "title": "Song title"
    }, 
    {
      "length": 205, 
      "id": 1, 
      "title": "Song title 2"
    }
  ]
}

After some thought, I think that supporting two formats is the best option.

1. Just raw list of urls or song titles which is meant to be created by users manually, imported and converted to JSON format.
2. JSON with meta info as suggested, but without that song_urls array.

When comparing JSON to m3u, I still find it easier to modify than m3u.

[ids1024]

The #EXTINF lines are optional, and part of Extended m3u rather than basic m3u. They start with the # comment character for backward compatibility. The user would only need to create a list of urls, and those lines could be added automatically.

An advantage of using a standard like m3u is compatibility with other software, though not much would support youtube urls. Mpv, however, can play an m3u like in my post just fine. This is not really important though.

Good idea for splitting out the metadata, but what if the user deletes an url or adds one to the middle? All metadata after that point would be messed up.

I think yaml rather than json is fine, except for the fact it pulls in an additional dependency. I don't really care, but some people might. But then, the yaml and json should both serialize straight to python objects, so it would be fairly trivial to support both, though that may cause more trouble than it's worth.

[Atlantic777]

Deleting an url could be made failsafe by copying URL into meta info.

• Playlist is created using just required part of JSON/YAML format playlist
• Playlist is imported and mpsyt generates meta data, and each meta data entry has it's URL.
• Both URL array and meta data are hashed, and hashes are stored as "last known good configuration"
• An entry is removed from URL array or from meta data section. Based on hash of section, we can determine which section changed and ask user what to do, to remove the entry from the other section, to undo change or to add new data (new meta entry, or new URL depending which one is missing).

This scenario is a bit outstretched and I wouldn't support it in early implementations. I would split this into 2 or 3 feature request/todo items. Similar problems raise when URL is changed in URL list or meta data.

I would once again emphasise that I think that it's the best to support raw URL list format just for import/export and generate JSON/YAML style playlists which are still easy to handle manually.

[ids1024]

It is rather important to handle removing of an url from the playlist. Perhaps storing playlists in a json format like the yaml in my post and allowing a simple url per line format for import/export as well would be best? I don't think the json format is all two bad for editing (yaml is perhaps a bit better for writing; less so for editing) to avoid adding the dependency on yaml.

I still think m3u is not a bad option either, but admittedly not the most readable. But then, it's not like the user would have to write the EXTINF lines...

[ids1024]

But anyway, this version of your yaml format should work better, by using urls rather than indexes. I don't see all too much advantage to yaml over json in this case though...

# required
title : my playlist

# required 
song_urls : [
"url0",
"url1",
"url2",
"url3",
"url4",
"url5",
]

# generated on first load
meta:
    "url0":                 # references songs url array
        title  : Song title # doesn't have to match video title
        length : 200        # just for matching purposes

    "url1":
        title  : Song title 2
        length : 205

Equivalent json:

{
    "title": "my playlist",

    "song_urls": [
        "url0",
        "url1",
        "url2",
        "url3",
        "url4",
        "url5"
    ],

    "meta": {
        "url1": {
            "length": 205,
            "title": "Song title 2"
        },
        "url0": {
            "length": 200,
            "title": "Song title"
        }
    }
}

Oh, as a side note, it could be better to use youtube ids rather than url. Any format meant to be written by the user, though, should handle both and convert as necessary.

[spcmd]

Any update on this? I think the config file should be plain text too.

[ikuraj]

This would be super useful to have -- a playlist that, in addition to being portable, can be diffed and edited as text.

[ids1024]

I can implement this once the format is agreed upon. Am I the only one who likes the idea of using m3u for playlists (which would be compatible with mpv as well, since it supports youtube urls, but not most software)? Any votes for m3u, json, or yaml (which adds a dependency)?

[ikuraj]

I think m3u, especially since it's a standard format for playlists, would work nicely -- my vote for m3u.

[kraetzin]

Adding my +1 for m3u

[ids1024]

One thing to note with m3u is that most other software won't play an m3u file with youtube urls, though I know at least mpv will handle it. That somewhat reduces the advantage of using a standard format, though I personally think it is the best choice.

Then there is the matter of the format for the configuration, which I think should be changed to a text format as well. Python's configparser format (a variant of .ini) is probably the best choice for that.

[PuKoren]

+1 for this feature and m3u, standard is good

Being able to use mps-youtube coupled with GitHub and its PR system to manage shared playlists would be awesome

[RockyRoad29]

Maybe json export is not that complex. We could serialize playlists with

import json
# ...
with open('/tmp/playlists.json', 'w') as out:
    json.dump(playlists, out, default=lambda x: x.__dict__)

Deserialization shouldn't be out of reach either.

[ids1024]

@RockyRoad29 Yeah, this is not really complicated to implement; it's just a matter of determining exactly what format to use. If the new format turns out to be a bad idea and is then replaced with something else, new releases will have to include converters for the format forever, which is annoying.

I still haven't gotten much response about the config format. What format do people think should be used for mpsyt's configuration? As mentioned above I'm leaning toward python's configparser format (basically .ini format). Json is also an option. I think I'd prefer to avoid yaml though, since it would add a new dependency to mpsyt.

[RockyRoad29]

I understand that such changes need to be thought carefully. I just wrote a quick-and-dirty exporter script for myself, starting from your snippet above, and thought it could be shared.

My two cents about format is that, to me, json seems to be the most popular format today for coders (webapp apis, js libraries ...). Yaml is easier to read for a human, but not for a program.

Thank you very much for mpsyt, I use it regularly.

[PuKoren]

@ids1024 no additional dep is great as I think mps-youtube is liked for its lightness, and .ini is very readable for both human and machines, so my :+1: for this

[leothelocust]

+1 for m3u and configparser

[rocketinventor]

@ids1024 I tried the code snippet that you gave but there seems to be an unmet dependency.

Traceback (most recent call last):
  File "playlists.py", line 7, in <module>
    playlist = pickle.load(file)
  File "/usr/lib/python2.7/pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "/usr/lib/python2.7/pickle.py", line 858, in load
    dispatch[key](self)
  File "/usr/lib/python2.7/pickle.py", line 1090, in load_global
    klass = self.find_class(module, name)
  File "/usr/lib/python2.7/pickle.py", line 1124, in find_class
    __import__(module)
ImportError: No module named mps_youtube.playlist

[ids1024]

python2.7

I don't know how you have mpsyt installed, but it only works on Python 3 currently, so try with python3.

[rocketinventor]

@ids1024 That seems to have been the issue. Thanks!

[rocketinventor]

I noticed that each song has some sort of number listed with it. What are these numbers?

[ids1024]

You mean the length (in seconds)?

[rocketinventor]

Oh, yes. Of course. 🤦