LS an Remote Relays not playing nicely. Radio glitching, can't keep up, dead air, restarts automatically

[alexhorner]

Describe the bug Adding remote relays causes LS to start spewing We must catchup X.XX seconds! (more relays = more seconds. 6 relays is about 4-10 seconds, 18 is about 40-50 seconds)

Instability including 20-30 second spans of dead air, glitchiness, half cut off jingles and songs etc start to occur the worse the catchup gets. Restarts only make the issue worse before it settles. DJs connecting will make this even worse and sometimes even kill the streams, causing LS to drop the relays.LS log can be found here https://gist.github.com/alexhorner/57c3bd5699719422a5c34d8441401e81

For more information, see AzuraCast/AzuraCast#3036

To Reproduce

output.icecast(%mp3(samplerate=44100, stereo=true, bitrate=320, id3v2=true), id="simulator_radio_relay_1", host = "atlas.simulatorradio.stream", port = 8000, user = "source", password = "passwordredacted", mount = "/stream", name = "Simulator Radio", description = "Your #1 Simulation Station", genre = "Varied", url = "https://simulatorradio.com/", public = false, encoding = "UTF-8", radio)
output.icecast(%mp3(samplerate=44100, stereo=true, bitrate=320, id3v2=true), id="simulator_radio_relay_2", host = "atlas.simulatorradio.stream", port = 8000, user = "source", password = "passwordredacted", mount = "/stream.mp3", name = "Simulator Radio", description = "Your #1 Simulation Station", genre = "Varied", url = "https://simulatorradio.com/", public = false, encoding = "UTF-8", radio)
output.icecast(%mp3(samplerate=44100, stereo=true, bitrate=320, id3v2=true), id="simulator_radio_relay_3", host = "kiros.simulatorradio.stream", port = 8000, user = "source", password = "passwordredacted", mount = "/stream", name = "Simulator Radio", description = "Your #1 Simulation Station", genre = "Varied", url = "https://simulatorradio.com/", public = false, encoding = "UTF-8", radio)
output.icecast(%mp3(samplerate=44100, stereo=true, bitrate=320, id3v2=true), id="simulator_radio_relay_4", host = "kiros.simulatorradio.stream", port = 8000, user = "source", password = "passwordredacted", mount = "/stream.mp3", name = "Simulator Radio", description = "Your #1 Simulation Station", genre = "Varied", url = "https://simulatorradio.com/", public = false, encoding = "UTF-8", radio)
output.icecast(%mp3(samplerate=44100, stereo=true, bitrate=320, id3v2=true), id="simulator_radio_relay_5", host = "alaska.simulatorradio.stream", port = 8000, user = "source", password = "passwordredacted", mount = "/stream", name = "Simulator Radio", description = "Your #1 Simulation Station", genre = "Varied", url = "https://simulatorradio.com/", public = false, encoding = "UTF-8", radio)
output.icecast(%mp3(samplerate=44100, stereo=true, bitrate=320, id3v2=true), id="simulator_radio_relay_6", host = "alaska.simulatorradio.stream", port = 8000, user = "source", password = "passwordredacted", mount = "/stream.mp3", name = "Simulator Radio", description = "Your #1 Simulation Station", genre = "Varied", url = "https://simulatorradio.com/", public = false, encoding = "UTF-8", radio)
output.icecast(%mp3(samplerate=44100, stereo=true, bitrate=64, id3v2=true), id="simulator_radio_relay_7", host = "kiros.simulatorradio.stream", port = 8000, user = "source", password = "passwordredacted", mount = "/64", name = "Simulator Radio", description = "Your #1 Simulation Station", genre = "Varied", url = "https://simulatorradio.com/", public = false, encoding = "UTF-8", radio)
output.icecast(%mp3(samplerate=44100, stereo=true, bitrate=96, id3v2=true), id="simulator_radio_relay_8", host = "kiros.simulatorradio.stream", port = 8000, user = "source", password = "passwordredacted", mount = "/96", name = "Simulator Radio", description = "Your #1 Simulation Station", genre = "Varied", url = "https://simulatorradio.com/", public = false, encoding = "UTF-8", radio)
output.icecast(%mp3(samplerate=44100, stereo=true, bitrate=128, id3v2=true), id="simulator_radio_relay_9", host = "kiros.simulatorradio.stream", port = 8000, user = "source", password = "passwordredacted", mount = "/128", name = "Simulator Radio", description = "Your #1 Simulation Station", genre = "Varied", url = "https://simulatorradio.com/", public = false, encoding = "UTF-8", radio)
output.icecast(%mp3(samplerate=44100, stereo=true, bitrate=192, id3v2=true), id="simulator_radio_relay_10", host = "kiros.simulatorradio.stream", port = 8000, user = "source", password = "passwordredacted", mount = "/192", name = "Simulator Radio", description = "Your #1 Simulation Station", genre = "Varied", url = "https://simulatorradio.com/", public = false, encoding = "UTF-8", radio)
output.icecast(%mp3(samplerate=44100, stereo=true, bitrate=64, id3v2=true), id="simulator_radio_relay_11", host = "atlas.simulatorradio.stream", port = 8000, user = "source", password = "passwordredacted", mount = "/64", name = "Simulator Radio", description = "Your #1 Simulation Station", genre = "Varied", url = "https://simulatorradio.com/", public = false, encoding = "UTF-8", radio)
output.icecast(%mp3(samplerate=44100, stereo=true, bitrate=96, id3v2=true), id="simulator_radio_relay_12", host = "atlas.simulatorradio.stream", port = 8000, user = "source", password = "passwordredacted", mount = "/96", name = "Simulator Radio", description = "Your #1 Simulation Station", genre = "Varied", url = "https://simulatorradio.com/", public = false, encoding = "UTF-8", radio)
output.icecast(%mp3(samplerate=44100, stereo=true, bitrate=128, id3v2=true), id="simulator_radio_relay_13", host = "atlas.simulatorradio.stream", port = 8000, user = "source", password = "passwordredacted", mount = "/128", name = "Simulator Radio", description = "Your #1 Simulation Station", genre = "Varied", url = "https://simulatorradio.com/", public = false, encoding = "UTF-8", radio)
output.icecast(%mp3(samplerate=44100, stereo=true, bitrate=192, id3v2=true), id="simulator_radio_relay_14", host = "atlas.simulatorradio.stream", port = 8000, user = "source", password = "passwordredacted", mount = "/192", name = "Simulator Radio", description = "Your #1 Simulation Station", genre = "Varied", url = "https://simulatorradio.com/", public = false, encoding = "UTF-8", radio)
output.icecast(%mp3(samplerate=44100, stereo=true, bitrate=64, id3v2=true), id="simulator_radio_relay_15", host = "alaska.simulatorradio.stream", port = 8000, user = "source", password = "passwordredacted", mount = "/64", name = "Simulator Radio", description = "Your #1 Simulation Station", genre = "Varied", url = "https://simulatorradio.com/", public = false, encoding = "UTF-8", radio)
output.icecast(%mp3(samplerate=44100, stereo=true, bitrate=96, id3v2=true), id="simulator_radio_relay_16", host = "alaska.simulatorradio.stream", port = 8000, user = "source", password = "passwordredacted", mount = "/96", name = "Simulator Radio", description = "Your #1 Simulation Station", genre = "Varied", url = "https://simulatorradio.com/", public = false, encoding = "UTF-8", radio)
output.icecast(%mp3(samplerate=44100, stereo=true, bitrate=128, id3v2=true), id="simulator_radio_relay_17", host = "alaska.simulatorradio.stream", port = 8000, user = "source", password = "passwordredacted", mount = "/128", name = "Simulator Radio", description = "Your #1 Simulation Station", genre = "Varied", url = "https://simulatorradio.com/", public = false, encoding = "UTF-8", radio)
output.icecast(%mp3(samplerate=44100, stereo=true, bitrate=192, id3v2=true), id="simulator_radio_relay_18", host = "alaska.simulatorradio.stream", port = 8000, user = "source", password = "passwordredacted", mount = "/192", name = "Simulator Radio", description = "Your #1 Simulation Station", genre = "Varied", url = "https://simulatorradio.com/", public = false, encoding = "UTF-8", radio)

The above is the remote relay configurations

Expected behavior LS should not struggle to handle multiple relays simultaneously

Version details

• OS: Ubuntu 18.04.4 LTS
• Version: 1.4.3 prerelease commit 13d9445

Install method AzuraCast

[toots]

Hi! Have you checked your CPU utilization?

[alexhorner]

CPU during this shows LS pinning 1 core of a 4 core machine to 100% whilst the other cores idle

[toots]

Ha I see. That's a known limitation of OCaml, the language that liquidsoap is written in. You can. however, work around it using clocks. There's some documentation at the bottom of this page: https://www.liquidsoap.info/doc-1.4.2/clocks.html, section "Internal clocks: exploiting multiple cores".

Also, one optimization that will be coming with the next major release is the ability to encode once and send to different destinations so all the relays with the same encoding params would be encoded only once.

Lastly, I would recommend to also look at icecast's configuration settings. It looks like some alternative mount points that you are using such as /stream and /stream.mp3 could be dealt with at the icecast level without having to send two streams for each.

[BusterNeece]

@toots Is it safe to just wrap every output in its own: clock.assign_new(sync=false,[output.icecast(...)]) call to take advantage of multi-core infrastructures without overloading a single core?

Update: if I simply assign a new clock a-la the example in the previous line, I run into this error:

At line 141, char 29-333:
Error 10: A source cannot belong to two clocks (wallclock_azuratest_radio_local_1[cross_7750[cue_cut_7742[],cue_cut_7730[],cue_cut_7717[]]],
wallclock_azuratest_radio_local_2[]).

...however, if I wrap the signal in mksafe(buffer(radio)) in the new clock, it plays maybe 3 seconds every minute and the rest is dead silence.

The log just constantly is filled with messages like this:

2020/07/17 05:59:21 [warp_prod_7801:3] Buffer emptied, start buffering...
2020/07/17 05:59:21 [mksafe:3] Switch to safe_blank with forgetful transition.
2020/07/17 05:59:21 [mksafe:3] Switch to warp_prod_7792 with transition.
2020/07/17 05:59:21 [warp_prod_7792:3] Buffer emptied, start buffering...
2020/07/17 05:59:21 [mksafe:3] Switch to safe_blank with forgetful transition.
2020/07/17 05:59:22 [mksafe:3] Switch to warp_prod_7801 with transition.
2020/07/17 05:59:22 [warp_prod_7801:3] Buffer emptied, start buffering...
2020/07/17 05:59:22 [mksafe:3] Switch to safe_blank with forgetful transition.

[toots]

@SlvrEagle23 Ideally you would have one clock per CPU core and assign outputs evenly to each of them. I think the situation to test in this case would be:

• Organise the outputs in 4 groups, one for each core, evenly.
• Attach a clock to each group using buffers like you described.

Then, I would check again on CPU usage. You might still be over the CPU capacity, in which case the symptoms that you describe should be expected.

Maybe you could also start with one output per clock (and per core), see if it works and then progressively ramp up the number of outputs?

[alexhorner]

@toots Would you be able to provide an implementation example where 4 or so outputs are attached to one input in the fashion you described using clocks? I too attempted myself to split it and achieved the same result as @SlvrEagle23 on the first round (error) and could not achieve the second round

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue was closed for lack of activity. If you believe that it is still relevant, please confirm that it applies to the latest released version of liquidsoap and re-open the ticket. Thanks!