Audio: UltraGrid has crashed (Segmentation fault)

[alatteri]

[2021-12-20 21:35:12] New incoming audio format detected: 48000 Hz, 8 channels, 32 bits per sample, codec AAC [2021-12-20 21:35:12] [lavcd aud.] Using audio decoder: aac [2021-12-20 21:35:12] Backtrace: [2021-12-20 21:35:12] [2021-12-20 21:35:12] /lib/x86_64-linux-gnu/libc.so.6(+0x46210)[0x7fed56333210] [2021-12-20 21:35:12] uv(decode_audio_frame+0x108)[0x55ce7cb58748] [2021-12-20 21:35:12] uv(_Z11pbuf_decodeP4pbufRKNSt6chrono10time_pointINS1_3_V212system_clockENS1_8durationIlSt5ratioILl1ELl1000000000EEEEEEPFiP10coded_dataPvP10pbufstatsESE+0x94)[0x55ce7cb53044] [2021-12-20 21:35:12] uv(+0x83586)[0x55ce7cb76586] [2021-12-20 21:35:12] /lib/x86_64-linux-gnu/libpthread.so.0(+0x9609)[0x7fed56834609] [2021-12-20 21:35:12] /lib/x86_64-linux-gnu/libc.so.6(clone+0x43)[0x7fed5640f293] [2021-12-20 21:35:12] [2021-12-20 21:35:12] UltraGrid has crashed (Segmentation fault). [2021-12-20 21:35:12]

[alatteri]

Good morning everyone and Happy New Year!!!

The above bug/crash is relatively new. It seems to only happen if the stream is already active when the receiver comes online. If the stream starts after the receiver is online, then this crash doesn't seem to happen.

Unfortunately, this is a blocking bug for us. I have tried rolling back to a variety of commits going back several months, so I don't think the issue is UG code specific, but maybe some sub-system of Ubuntu 20.04 LTS has updated and changed. How can I help get you the information you need?

[MartinPulec]

Hi Alan, thanks and happy New Year to you as well!

As for the bug:

1. what is the exact (or minimal) configuration (arguments) for the bug
2. is it occurring also for different codecs (OPUS, PCM)?
3. assuming that the problem is in AppImage, would it be possible try to the source-code build?
4. which is the exact version of the AppImage, and/or would it be possible to re-run (and paste output) it with some defined version (eg. this)? It will help parsing the exact line from the stacktrace, I'd be especially interested in the exact line in decode_audio_frame.

Points 2, 3, 4 may or may not be needed, depending if we'll be able to reproduce the issue.

It may or may not be in UltraGrid - the thing that it occurs now but not before may also mean, that some recent change has triggered the bug.

[alatteri]

Hi Martin,

1. So far I have only tested with 8 channel Decklink embedded SDI source
2. I've only tried OPUS and ACC
3. & 4. These are source code builds, but I think I've also tried AppImage.

I don't believe the issues itself lies in UG code. I've going way back to commits that very likely worked before, like 6-8 months ago. I feel like one of the audio libraries changed in Ubuntu with recent updates, even though I am using 20.04 LTS, and that is causing the crash.

[MartinPulec]

Ok, so please let me know if you find something out. Although I cannot determine exact point of failure in the function decode_audio_frame(), it seems suspicios that it is inside some libc function. I've read through the function and it seems to me that just memcpy matches, which could indicate something nasty.

I may try using valgrind to see if I find something. But otherwise I am afraid that I've currently few information.

[alatteri]

What would be the best debug mode to run in to give more information?

[MartinPulec]

I'd suggest the following kick-off:

1. a minimal working configuration - eg. does this exhibit even with one-channel audio testcard + let say dummy receiver? what is then the minimal sender/receiver cmdline args which does exhibit the problem for you
2. running some defined UltraGrid instance (in our case perhaps AppImage) and re-pasting the output here - I cannot map the stacktrace addresses to source code lines without having the executable (well debug its symbols should suffice) - so alternatively sending the executable will be also ok

UPDATE: I'd pehaps need more info, indeed - I've tried to run a simple case with valgrind:

 valgrind uv -r dummy    # this run first
 uv  -s testcard --audio-codec AAC --audio-capture-format channels=8

(the second command needs today's fix to run)

which didn't exhibit the problem to me.

[alatteri]

I'm closing this as this doesn't seem to be happening anymore.