kh22 Icecast crashes

[anisakharov]

Hello! We updated to Linux version 22, that is, the latest. We have several servers with an average number of listeners of 10-13k listeners. Periodically, Icecast crashes, it hangs in processes, but no connections are made. There is nothing unusual in the logs - only old errors before the crash. Is there any other way to determine why the icecast drops? Thank you in advance for your cooperation!

[Csaba27]

I have same problem (ububtu 22.04), i cannot debuging what causing the problem. For me, it can stop when the stream is not .mp3 and the broadcast close sometimes. I tried many versions but i don't know why crashing.

[anisakharov]

We also have backup mount points, aac+ format. But both the backup points and the original ones are absolutely identical in format and other parameters. We still cannot understand that this is the reason. Karl needs help! ))

[karlheyes]

sorry for not getting back sooner, had non-icecast issues to deal with. Obviously need to know where the hang is in such cases, do make clean debug and run to get a hang case. Then use gcore to grab a core file which can be inspected

karl.

[anisakharov]

The point is that it is impossible to understand when an icecast crash will occur. This can happen every day or every 3-5 days. If you enable debug logs, will the data be displayed or not? How to gcore?

[anisakharov]

The only thing we noticed was a problem with determining the aacp format. For example, we receive an aacp stream from another server. If you do not specify the type, aacp is automatically displayed. However, if the current icecast relays the stream, then it itself produces the aac type. As I understand it, aac and aacp are different formats and perhaps this is the problem.

[anisakharov]

sorry for not getting back sooner, had non-icecast issues to deal with. Obviously need to know where the hang is in such cases, do make clean debug and run to get a hang case. Then use gcore to grab a core file which can be inspected

karl.

?

[patphobos]

Hi Karl,

Just got this crash on a test server. mount point has no auth. Mount point in an on-demand relay (with multiple relays). I'm currently trying with on-demand=0 instead of 1, it seem more stable. I cant reproduce the crash manualy, i need to put the server in "test-production" to crash it.

Best

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/home/patrice/icecast-kh/src/icecast -b -c /etc/icecast2-kh/icecast-streamweb.x'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f6dee718ef4 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libc.so.6
[Current thread is 1 (Thread 0x7f6deb4aa640 (LWP 4026608))]
(gdb) backtrace full
#0  0x00007f6dee718ef4 in pthread_mutex_lock () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#1  0x000055eaffac36e8 in _mutex_lock_c (mutex=0x10, file=0x55eaffacdb10 "source.c", line=2922) at thread.c:845
        rc = <optimized out>
#2  0x000055eaffa9c357 in source_setup_listener (source=source@entry=0x7f6dac216140, client=client@entry=0x7f6db0003ef0) at source.c:2922
        worker = 0x0
        __func__ = "source_setup_listener"
#3  0x000055eaffa9d1ee in source_add_listener (mount=<optimized out>, mount@entry=0x7f6de008c1c0 "/stream/natio_mp3_128k", mountinfo=mountinfo@entry=0x55eb0042d6a0, client=client@entry=0x7f6db0003ef0) at source.c:2873
        loop = <optimized out>
        rate = 0
        do_process = 0
        within_limits = <optimized out>
        source = 0x7f6dac216140
        minfo = <optimized out>
        passed_mount = 0x7f6de008c1c0 "/stream/natio_mp3_128k"
        len = 0
        config = <optimized out>
        max_bandwidth = 0
        max_listeners = 0
        __func__ = "source_add_listener"
#4  0x000055eaffab4350 in add_authenticated_listener (mount=mount@entry=0x7f6de008c1c0 "/stream/natio_mp3_128k", mountinfo=0x55eb0042d6a0, client=client@entry=0x7f6db0003ef0) at auth.c:540
        ret = 0
        __func__ = "add_authenticated_listener"
#5  0x000055eaffab57a4 in auth_add_listener (mount=mount@entry=0x7f6de008c1c0 "/stream/natio_mp3_128k", client=client@entry=0x7f6db0003ef0) at auth.c:741
        ret = 0
        need_auth = <optimized out>
        mountinfo = <optimized out>
        __func__ = "auth_add_listener"
#6  0x000055eaffa8bc49 in _handle_get_request (client=0x7f6db0003ef0) at connection.c:1898
        serverhost = <optimized out>
        serverport = <optimized out>
        ret = 0
        alias = 0x0
        config = 0x55eaffae5a00 <_current_configuration>
        uri = 0x7f6de008c1c0 "/stream/natio_mp3_128k"
        client_limit_reached = 0
        __func__ = "_handle_get_request"
        client_limit = 80000
#7  0x000055eaffaa349a in worker (arg=arg@entry=0x55eb001a4ff0) at client.c:876
        nxc = 0x0
        ret = <optimized out>
        client = 0x7f6db0003ef0
        worker = 0x55eb001a4ff0
        prev_count = 1
        wc = {worker = 0x55eb001a4ff0, prevp = 0x55eb001a5058, sched_ms = 1709726300410, wakeup_ms = 1709726330408, max_run = 999, time_recheck = 49, flags = 0}
        __func__ = "worker"
#8  0x000055eaffac3871 in _start_routine (arg=0x55eb001a50e0) at thread.c:768
        start = 0x55eb001a50e0
        start_routine = 0x55eaffaa3310 <worker>
        real_arg = 0x55eb001a4ff0
        thread = 0x55eb001a50a0
#9  0x00007f6dee715ac3 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.
#10 0x00007f6dee7a7850 in ?? () from /lib/x86_64-linux-gnu/libc.so.6
No symbol table info available.