savonet / liquidsoap

Liquidsoap is a statically typed scripting general-purpose language with dedicated operators and backend for all thing media, streaming, file generation, automation, HTTP backend and more.
http://liquidsoap.info
GNU General Public License v2.0
1.41k stars 130 forks source link

Liquidsoap randomly stopping #1717

Closed hydrosIII closed 3 years ago

hydrosIII commented 3 years ago

Hi I am running liquidsoap in a server that is receiving input from an icecast server from the internet, and outputting to alsa. The server is connected to an FM transmitter.

Server is debian 10.10 fully updated. Liquidsoap 1.4.4 installed via opam fully updated.

liquidsoap config file:

set("server.telnet.bind_addr", "127.0.0.1")
set("server.telnet.port", 9876)
set("server.telnet", true)

set("log.file",true)
set("log.file.path","/home/violetaradio/violeta.log")
set("log.stdout",true)
set("log.level",5)
#set("debug_level",5)
#set("decoder.debug", true)
#set("ffmpeg.log.verbosity", "debug")

### se aumento el buffer por errores que estaba teniendo 28 junio 2021
radio = input.https( buffer=10.0, max=100.0, "https://flujos.mazorca.org/violetaradio.mp3")
output.alsa(id="alsa", mksafe(radio))
#output.dummy(id="DUMMY_STREAM_OUTPUT", fallible=true, input_https)
#output.alsa(id="alsa", radio, fallible=true)

liquidsoap is running via a sytemd user service, and it restarts on error. However sometimes liquidsoap just hangs without any error at a random time. The result is that the FM radio stops. To circunvenct this issue i am restarting liquidsoap via cron every hour. But the error sometimes it is there. But i got nothing in the logs. I can just see the blank time space in the logs:

Captura de pantalla de 2021-06-28 15-49-09

Check for example the timeframe betweetm 12:37 and 13:23 in the screenshot. At 13:23 the cron job kicked in, as you can see Log level is at 5, tha maximum

Consequently i don't get sound from the time liquidsoaps hangs until the cron job restarts it. I have to kill liquidsoap, so systemd detects it at restarts it .

I restart it using a simple bash script with the kill command, every hour.

killall -9 liquidsoap

There was a live program (talk show ) when this happened, so no music tracks. But a long voice track (?). Does this matters? Network was stable, so it was not a network hiccup. Also the original icecast server was also stable.

StudentSA commented 3 years ago

Out of interest any specific reason you want to do this with liquidsoap? Have you considered a straight forward ffmpeg process instead?

hydrosIII commented 3 years ago

No specific reason, It was the tool that I found. After testing a few terminal media players I figured out liquidsoap was the way to go, because its simplicity and wide usage. But as you say, maybe there is a more simple form to do it.

hydrosIII commented 3 years ago

Ok. I am now running a simple ffplay process, powered with systemd. However now that i remember one of the reasons to run liquidsoap was to turn to a backup playlist when the internet was not available. Not implemented yet. But, how about the bug?

toots commented 3 years ago

Hi @hydrosIII. Thanks for reporting. We would love to fix this bug as part of our 2.x release cycle. Would you be interested in trying the latest main code?

My first suggestion would be to use alsa in unbuffered mode. There are some explanations on how to do so at the bottom of out cookbook page over at: https://www.liquidsoap.info/doc-dev/cookbook.html

hydrosIII commented 3 years ago

Thanks, @StudentSA, I used ffplay and now it is worst !!! It stopped but the program said that it was running !!!! I kind of remember i got similar behaviour in other audio players. So back to liquidsoap, at least i had a trace in the logs, and when it hangs I know it is hanged. Thanks toots, I am moving to the latest code.

hydrosIII commented 3 years ago

I got [alsa:2] Underrun! You may minimize them by increasing the buffer size. 2021/07/01 11:14:38 [alsa:2] Trying to recover.. in the logs. However i didn't get lke in the example: Could not set buffer size to 'frame.size' (1920 samples), got 2048.

Also my buffer is already at 100 max.

hydrosIII commented 3 years ago

I updated to liquidsoap latest main. Liquidsoap 2.0.0+git@3c254488 No I cannot start it because input is not aceppting http variable. Error 5: this value has no method http

toots commented 3 years ago

@hydrosIII you need to install the ffmpeg package. We have a page to help with migrating from 1.4.x: https://www.liquidsoap.info/doc-dev/migrating.html

hydrosIII commented 3 years ago

I have : ffmpeg version 4.1.6-1~deb10u1 Copyright (c) 2000-2020 the FFmpeg developers built with gcc 8 (Debian 8.3.0-6) as you see I have been using it for ffplay. I think it may be some error in the compiling, I compiled it with OCAML 4.08.0 and using opam pin add liquidsoap .

hydrosIII commented 3 years ago

Ah, sorry ! the ocam ffmpeg package. Give me a minute. For anyone seeing this.

opam install ffmpeg

after installing dependencies with apt

hydrosIII commented 3 years ago

Ok, I am getting: jul 01 12:03:10 violetaradio liquidsoap[16333]: 2021/07/01 12:03:10 [alsa:2] Underrun! You may minimize them by increasing the buffer size. jul 01 12:03:10 violetaradio liquidsoap[16333]: 2021/07/01 12:03:10 [alsa:2] Trying to recover.. but I cannot increase the buffer, since now the parameter buffer is not accepted for the input.http method. Ok, no. It gets stuck there.

hydrosIII commented 3 years ago

I finally got it started with: Some values and parameters changed. Also i added the frame audio size for alsa and my specific hardware, and used unbuffered alsa. I couldn't add set("frame.video.rate", 0) since the value is not valid anymore.

set("log.file",true)
set("log.file.path","/home/violetaradio/violeta.log")
set("log.stdout",true)
set("log.level",5)

set("frame.audio.size",1881)
radio = input.http( max_buffer=100.0, "https://flujos.mazorca.org/violetaradio.mp3")
output.alsa(bufferize=false,id="alsa", mksafe(radio))

It is running, I hope it stays like that

toots commented 3 years ago

I couldn't add set("frame.video.rate", 0) since the value is not valid anymore.

Are you sure that you are on the latest main? It should be valid.

toots commented 3 years ago

The doc about unbuffered alsa is at the bottom of this page: https://www.liquidsoap.info/doc-dev/cookbook.html

toots commented 3 years ago

Ha my bad, wrong key it should be:

set("frame.video.framerate", 0)

Fixing the doc.

hydrosIII commented 3 years ago

Thanks for changing the doc. So far so good it is running. But let me test for about a month if it runs stabl,e before declaring it closed, sometimes it takes time for the bug to appear. Also and maybe unrelated, I am getting some underruns:

jul 03 15:18:27 violetaradio liquidsoap[28593]: 2021/07/03 15:18:27 [alsa:2] Underrun! You may minimize them by increasing the buffer size. jul 03 15:18:27 violetaradio liquidsoap[28593]: 2021/07/03 15:18:27 [alsa:2] Trying to recover..

But it recovers, any idea how to make them go away?

toots commented 3 years ago

Glad to hear that it's moving forward. No, I don't know. This script is also dependent on your hardware. It's also possible that these issues are linked to some underlying hardware issues.

hydrosIII commented 3 years ago

Ok,I will try to acquire new hardware. Is there a database of hardware for sound cards for linux? Something the liquidsoap teams recommends? It has been running stable for almost a week. However it stops sometimes because of buffer issues and also i got:

Partial write (463 instead of 1881)! Selecting another buffer size or device can help. :4] Partial write (470 instead of 1881)! Selecting another buffer size or device can help. :4] Partial write (477 instead of 1881)! Selecting another buffer size or device can help. :4] Partial write (485 instead of 1881)! Selecting another buffer size or device can help.

But it recovers succesfully from them at seconds from this. Also the buffer is already at 100 seconds in the input, so as you say it is probably a hardware/driver issue. I am closing this, i hope it stays stable, but everything indicates it is going to be stable. Besides i am implementing a silence detector also wiith liquidsoap to restart it in case of failure.

hydrosIII commented 3 years ago

Further question, is this bot caused by using alsa unbuffered?

hydrosIII commented 2 years ago

My answer to my previous question. Yes I needed some buffer in the alsa output and the issue of thje partial write dissappears