Streaming MP3 pauses when server usy

[whatdoineed2do]

I have observed when streaming MP3 to a couple of devices and if the server becomes busy the streaming pauses

This is seen when streaming to Wifi or wired devices so it's not network latency. In fact I've observed this when both wifi/wired are connected simultaneously and both pause and restart at the same time

Is there an area where this can be investigated such as priority of read/transcode etc?

[whatdoineed2do]

"Busy" is like when we queue a large number of tracks (all for a genre for instance) to the play queue

[ejurgensen]

When you make the queue request the httpd thread will work exclusively on handling that until the response has been returned. During that time no mp3 will be served to the streaming clients, so if it takes a long time and/or the client has a small buffer, it will underrun.

To fix this it would be necessary to change the httpd architechture, so that all work is done in other threads. Or, a little bit more simple, would be to move the streaming part to have its own server running in its own thread (which would mean that the stream could no longer be served on port 3689). I did consider the latter when I first made the mp3 streaming, but I didn’t really like the idea of separate ports and yet a libevent loop.

[whatdoineed2do]

Regarding the httpd thread architecture, would you be open to a change to that? In particular having the ability for the httpd main inbound processing loop delegate requests off to a dedicated worker thread (ie json requests go to a dedicated worker thread etc).

I've put something together that allows for this. https://github.com/whatdoineed2do/forked-daapd/tree/mt-httpd (see httpd.c)

The httpd_init() can create workers for request types (json, streaming etc) if desired. The libevent callback to httpd_gen_cb delegates the specific request to the worker via a requests queue and returns meaning there is minimum blocking. Meanwhile, the worker thread will pick up the request and execute as normal and sends data back to the originating client via the same libevent loop.

Obviously this means any slow/long running workloads don't impede the http event loop from receiving or sending replies (like the mp3 streaming).

Whilst this works. I havent been able to find any documentation to confirm whether evhttp_send_reply() is thread safe which is what this solution relies on.

I've tested this by starting a streaming mp3 session and then calling a JSON endpoint that simply sleeps for 10 secs. The mp3 stream is uninterupted. Also tested using the web api whilst streaming and everything looks fine.

[whatdoineed2do]

I've been testing another implementation that tries to solve this - the trouble that I have is that my streaming client is a mostly a decade old Roku Soundbridge that seems to suffer from delays on the streaming.

httpd creates another event_base and evhttpd that is purely for handling streaming clients. The port is configurable and sent to the web UI as part of /api/config.

The main change (apart from the parallel setup of bindinng to ports etc) is in httpd_gen_cb which becomes a template method which delegates the functionality (daap/dacp/streaming etc) to a specific handler for the different types of httpd server we have: in this implementation there is 2: one that handles the streaming_request() and another for everything else.

Would this be acceptable for addressing this original issue even though it creates another event loop and mp3s are now served on a different port (not 3689)?

[whatdoineed2do]

FWIW, running this patched server on a RPI3, streaming to a laptop top -H -p $(pidof forked-daapd) shows the http thread for streaming at a constant 30-35%. When browsing all albums from the same laptop, the normal httpd thread jump to 65%.

Without the additional event loop as per the fixes, the streaming has a 2-3 pause.

[ben-willmore]

This might be of interest: I got libshine working as the streaming mp3 encoder, and it reduces the CPU usage on a Raspberry Pi quite a bit. On my Pi Zero, the httpd thread is down to 26-27% when streaming at the default 128kbps and quality. At this level, it's hard to produce any significant pauses or glitches in the stream, even without your separate http thread. I can browse/search a library of 1000 albums through the web interface without any noticeable glitches in the stream.

My edits are at https://github.com/beniamino38/forked-daapd/tree/mp3_encoder . To use libshine, set:

general {
  mp3_encoder = "libshine"
}

[whatdoineed2do]

This sounds interesting. This is without the compression_level = 9 workaround too.

The ffmpeg docs do say libshine is peformamce focus rather than sound quality focused which is a little bit of downer for me - I am streaming from a rpi3 to a roku soundbridge which feeds into a dac which was my motivation to get cfg'able MP3 quality working.

But I don't believe it's the MP3 Xcode that is the problem but rather the db query as initiated from json.

For me it was difficult to get a 100% reproducible set of steps to get a busy db/json subsystem (editing queue on web ui was best bet) but I replicated the delay by adding a new json endpoint that would just sleep N sec before returning. When N >2-3secs (as spec by query param) then the single http event loop server would result in a pause/gap on the MP3 stream. The dedicated http server for MP3 streaming is immune to this kind of blocking