Bad handling of large queue requests

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Select a lot of music in the ‘Folders’ tab
2. Try to add it to the play queue

What is the expected output? What do you see instead?
I expect everything selected to be added to the queue. Instead, nothing happens 
for few seconds and then an error message is shown: ‘Failed to send command 
to "yuuki-internet" <my-host-ip> - not connected’.

It works fine for smaller requests.

What version of the product are you using? On what operating system?
Latest SVN – Cantata 1.2.50
Linux misaki 3.12.3-gentoo #2 SMP Sat Dec 7 23:14:57 GMT 2013 i686 Intel(R) 
Core(TM)2 Duo CPU L7700 @ 1.80GHz GenuineIntel GNU/Linux

Please provide any additional information below.
The server doesn't seem to be spiking in load which suggests it's not getting 
any information about this queue-up. My wild guess is that Cantata thinks it 
times out while it's still building the request locally but I haven't peeked at 
the source. The MPD server is on a remote machine.

I'm trying this out with about 5000-9000 folders.

Original issue reported on code.google.com by fuuze...@fuuzetsu.co.uk on 29 Dec 2013 at 1:50

[GoogleCodeExporter]

Cantata waits for 4 seconds for the command to be written, and 8 seconds for 
the response from MPD. I'm guessing in your scenario, with such a long distance 
(and unusually large data), that one of these is timing out - but not sure 
which!

I've now modified the code, so it will use a timeout of 2 seconds per 100K of 
the message length. Can you please update trunk, and see if this helps?

Original comment by craig.p....@gmail.com on 29 Dec 2013 at 2:44

• Changed state: Started

[GoogleCodeExporter]

Same problem. Are there any debug flags I could set to get a log of some kind 
with what's happening?

Original comment by fuuze...@fuuzetsu.co.uk on 29 Dec 2013 at 5:00

[GoogleCodeExporter]

Hmmm... I've added more debug. Please re-compile, and then start Cantata (from 
the build folder) as:

CANTATA_DEBUG=1 ./cantata

This will then log MPD communications to ~/.cache/cantata/cantata.log

Please post the contents of this after the error occurs.

After you have done this, all I can suggest is to change 
'constSocketCommsTimeout=2000;'  to a higher value. The value is in 
milliseconds. If you do this, what value works?

Original comment by craig.p....@gmail.com on 29 Dec 2013 at 6:25

[GoogleCodeExporter]

You'll excuse the fact that I'm not going to post the whole 43k lines or so 
here. I'm hosting the whole thing at [1]. It's ~6.5MB. I have altered the 
server IP in the log to 000.000.000.000.

[1]: http://fuuzetsu.co.uk/misc/cantata.log

I will experiment with the timeout setting but I don't think it will be in a 
reasonable range and definitely not scalable: there's always someone with a 
bigger library and even setting it to 30000 does not help.

Does MPD not provide more efficient means of transfer? I'm literally selecting 
my top level directories in Cantata (picture related), is it not able to simple 
send ‘add everything under these few directories’?

Original comment by fuuze...@fuuzetsu.co.uk on 29 Dec 2013 at 11:49

Attachments:

• directories.png

[GoogleCodeExporter]

Just did some further playing with the timeout value and even if I set it to 
like 10 minutes, Cantata drops connection to the server as soon as usual (few 
seconds).

Original comment by fuuze...@fuuzetsu.co.uk on 30 Dec 2013 at 12:29

[GoogleCodeExporter]

Please update trunk, and try again! This time I've changed the code to send the 
list of files in chunks of 2000 filenames. If this does not work, can you 
change "constMaxFilesPerAddCommand=2000" (in mpd/mpdconnection.cpp) to a lower 
value? If it does work, can you increase the value in (e.g.) steps of 500 until 
it fails? I know the value reached will not be optimum - but if your case works 
for 5000 entries, then this migh tbe a better value for me to use...

Original comment by craig.p....@gmail.com on 30 Dec 2013 at 9:52

[GoogleCodeExporter]

I've now modified the code again - this time I've added a manual config item to 
set the chunk size. You need to set "mpdListSize" See the README for details.

Original comment by craig.p....@gmail.com on 30 Dec 2013 at 10:00

[GoogleCodeExporter]

mpdListSize=10000 (and smaller) works although it seems I'm hitting some MPD 
limits:

MPD reported the following error: playlist is at the max size

A quick search does say that I can increase max_playlist_length in MPD so 
that's fine. I think that with the mpdListSize option, we can consider this 
issue as solved. Thanks!

Original comment by fuuze...@fuuzetsu.co.uk on 30 Dec 2013 at 12:08

[GoogleCodeExporter]

Fwew! Thanks. I'll change the default to 10000

Original comment by craig.p....@gmail.com on 30 Dec 2013 at 12:13

• Changed state: Verified