Slow start with long playlist

[quite]

I have a 10000+ entries playlist, which makes sonata take a long time to start up (I couldn't wait, just kill -9ed it...)

[multani]

Thanks for the bug report!

By "playlist", did you mean the "Current" playlist. I tried to reproduce the bug on my computer with a 12000+ playlist and Sonata becomes very sluggish after that, but I don't have any problems with other playlists with that much number of songs inside.

I already have several ideas to speed this up and I'll make a branch. That would be really appreciated if you can give me some feedback about it!

[multani]

So I made a branch named long-playlist with some fixes for this problem.

In this branch, Sonata now loads the songs in a asynchronous-fashion, and doesn't freeze the whole UI while updating the list. It already looks better than before, but there are still things I need to do before considering this ready for master:

• there's no progress indicator of the loading process. If you have, say 10000+ songs in your playlist, there are going to be loaded progressively but you don't have any way to know how long it's going to take before finishing.
• there are probably tons of ways to break the UI now. Freezing everything was a good way to ensure nothing was happening during the loading process. Now, this assumption is not quite true anymore, and I'll need to poke around the code to see race-conditions while accessing the data...

Still, you can already give it a go and see if it at least eases your original problem.

[quite]

I tried the long-playlist branch now, but can't seem to get it working at all. I get a repeated message:

[2013-11-07 13:43:25] sonata.mpdhelper: Already connected

Some issue with mpd 0.18?

The current master works, but yes initial loading of the "current" playlist takes long time. I tried quickly to merge the diff HEAD^ from long-playlist onto master, but it was reject.

I'm a bit confused as to why this is so slow. Does modern graphical gui toolkits still have a problem to handle "large" amounts of items in listviews? That can't be... Is it due to some lookups through mpd?

In ncmpcpp, my 14k item "current" playlist loads in a second.

[multani]

The issue about being "Already connected" is due to MPD 0.18, I merged a fix in master a few hours ago, I'm going to rebase the branch against it.

As for the performance problem... I'm not exactly sure where the real bottleneck is. I'm far from being a GTK Guru but it looks like feeding the store where the listview takes its data to display the "Current" playlist already takes a long time. My branch tries to feed the model by batches, while allowing the view to refresh every once in a while. I've been poking other places in the code last week-end to see where I could have more performances and there are places which can be better although it impact seems minimal. I'm still considering other ways...

I'm curious how the data is loaded in ncmpcpp though. You said it loads in a second, but are you sure it loads all the data instantly or just the (small I guess ?) part that you are viewing?

[quite]

Well, mpc, ncmpcpp and sonata all seem to do plchanges "0" or playlistinfo to get the current playlist on startup. As far as I understand from the verbose mpd log, this is the single only command needed to get all info about all songs. ncmpcpp is quick, mpc quicker. But sonata unfortunately takes 9 seconds before it opens it's interface.

mpc playlist --format "%artist% - %title% -- %album% %track% %date% %time%"  0.41s user 0.10s system 64% cpu 0.788 total
wc -l  0.00s user 0.01s system 0% cpu 0.788 total

It guess this must be due to the way the gtk listview is built. I guess the core of the listview in ncmpcpp is much more simple and efficient. But there ought to be some way to achieve this speed in gtk as well.

[multani]

I also (tried) to have a look at gmpc and they are also building the list normally fast.

I made a quick test with some of Sonata's code, which basically looks like this:

    t = Timer()

    m = sonata.mpdhelper.MPDClient()
    m.connect('localhost', 6600)
    t.mark("Connect")
    songs = m.plchanges(0)
    t.mark("Fetch")

    l = Gtk.ListStore(sonata.mpdhelper.MPDSong)
    for song in songs:
        l.append((song, ))
    t.mark("Append")

Results are quite interesting. With current's Sonata, loading a 12000 songs playlist takes that long (times in seconds)

Connect: 0.023736953735351562
Fetch: 4.272056341171265
Append: 4.952490568161011

I ported a few lines of code to Python 2 and pygtk, and I've got the following results:

Connect: 0.00244903564453
Fetch: 8.22899985313
Append: 1.48366594315

Actually, Sonata does something more along this way:

    columnformat = ['%N', '%T', '%A', '%B', '%Y', '%L', '%Y - %B - %N']
    current_columns = [sonata.mpdhelper.MPDSong] + [str] * len(columnformat) + [int]

    t = Timer()

    m = sonata.mpdhelper.MPDClient()
    m.connect('localhost', 6600)
    t.mark("Connect")
    songs = m.plchanges(0)
    t.mark("Fetch")

    l = Gtk.ListStore(*(current_columns))
    for song in songs:
        items = [sonata.formatting.parse(part, song, True) for part in columnformat]
        l.append([song] + items + [Pango.Weight.NORMAL])
    t.mark("Append")

In this case, I get the following more realistic numbers:

Connect: 0.025308609008789062
Fetch: 4.345914602279663
Append: 37.98859691619873

[multani]

So I guess, there are rooms for improvement, in Sonata, python-mpd2 and PyGObject as well I guess.

(I tried the iterate mode of python-mpd2, but it doesn't really change anything for the final timing. Fetch is of course much faster, but the time just shifts to Append actually).

[multani]

Just to post a follow-up on this, I'm currently working on improving the speed of the Current playlist.

On a 3000+ playlist, it currently goes from about 12 seconds to less than 2 seconds. I'll post a branch when there's something interesting to test.

[multani]

I just pushed a branch at https://github.com/multani/sonata/tree/fix/faster-playlist-loading

It took about 45 seconds+ before to load a 13000 items playlist, it now takes about 9 seconds. Scrolling might be a bit slower, I would be interested to hear about it, but any other comments is also welcomed! The diff is quite large, so there might be some bugs that I missed, let me know!

[quite]

Oh, nice work. Yes it's really approaching usability now :) around 10 seconds for my 16k+ playlist. Maybe II'm silly, I just continue to add things to it, and never really want to clean it, because I like to go back and see what I listened to, and have quick access to that. (by the way, I saw & " etc in the playlist now)

[quite]

Scrolling seems all right still. Both mouse drag and pageup/down keys.

[multani]

OK, that's great! I'm going to fix the "&" issue, I'm also seeing it here. As for more speed, the biggest contender is now python-mpd2 (@Mic92 !). With the following code (which is roughly what Sonata does now to fill the ListStore):

t = Timer()

store = Gtk.ListStore(mpdhelper.MPDSong)
m = mpd.MPDClient()
m.connect('localhost', 6600)
t.mark("Connect")

songs = m.plchanges(0)
t.mark("Fetch")

songs = [mpdhelper.MPDSong(s) for s in songs]
t.mark("Convert")

for song in songs:
    store.insert_with_valuesv(-1, [0], [song])
t.mark("Insert")

I get the following results (with a 16000+ playlist)

Connect: 0.022413015365600586
Fetch: 5.222683668136597
Convert: 2.111999988555908
Insert: 1.2015612125396729

I'm going to see where I can start from this, if I missed something from python-mpd2 or if I can improve the performance there, or if I can somehow mitigate things in the middle...

@quite: by the way:

$ mpc playlist| wc -l
16384
$ mpc add /          
error: playlist is at the max size

;)

[multani]

I pushed an updated version with the fix for the markup.

[multani]

For reminder: I was going to merge this branch to master, but sorting the Current playlist is broken and need to be fixed.