Crash in circlebuf_peek_front during obs_source_output_audio

VictorKruger commented 1 year ago

Operating System Info

Other

Other OS

Linux Mint 21 Cinnamon

OBS Studio Version

28.0.3

OBS Studio Version (Other)

No response

OBS Studio Log URL

https://obsproject.com/logs/yMvMEwcRm3V2u_mF https://obsproject.com/logs/Lnk8HfTB4pMAP0W9 https://obsproject.com/logs/bIVWQ-oPx7I1B9pD

OBS Studio Crash Log URL

No response

Expected Behavior

To not close down and/or crash during a stream

Current Behavior

OBS stalls out for about 10 seconds and then closes without any error message.

Steps to Reproduce

1.Start streaming on twitch 2.stream for about an hour 3.OBS crashes ...

Anything else we should know?

I use the current version of obs teleport and I compile and use the current version of obs multi rtmp from github.

gxalpha commented 1 year ago

Please remove all third-party plugins and try again. Plugins can definitely crash OBS.

Also please include a crashlog.

VictorKruger commented 1 year ago

I definitely understand removing plugins to eliminate crashes due to plugins, and I can definitely remove the obs multi rtmp. However I'm currently using a two pc setup in which I use the teleport plugin to send the video and audio to the streaming pc.

I'm not sure if this will provide an accurate test, however I can remove the plugins on the streaming pc, start up a test streaming using some simple gpu tests like glxgears and vkcube and let it sit for a couple hours.

On Sun, Oct 9, 2022 at 12:11 PM Sebastian Beckmann @.***> wrote:

Please remove all third-party plugins and try again. Plugins can definitely crash OBS.

— Reply to this email directly, view it on GitHub https://github.com/obsproject/obs-studio/issues/7574#issuecomment-1272587063, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQPKLRQZ2UQ2ERPOPE27PFTWCL4FNANCNFSM6AAAAAARAYRHHQ . You are receiving this because you authored the thread.Message ID: @.***>

kkartaltepe commented 1 year ago

Provide a backtrace from GDB if you think obs is crashing please.

VictorKruger commented 1 year ago

Thank you for the quick responses.

@gxalpha called it, the crashes were due to a plugin, more specifically the latest version of obs-teleport. I will head over their github and see if I can compile it by source as there was a change done to the files last month or send a trouble ticket.

fzwoch commented 1 year ago

As a follow up. the crash was caused by Teleport, but not within itself. It was triggering an assert inside OBS:

obs: ./libobs/util/circlebuf.h:246: circlebuf_peek_front: Assertion `size <= cb->size' failed.

So potentially Teleport tried to push data with 0 size? Hopefully with an updated Teleport this will not occur again.

Some remaining questions for me are:

Is it intentional, that release versions of OBS do assert? No NDEBUG flag being set?
Is that assert behavior on all platforms the same or is this on Linux only?

kkartaltepe commented 1 year ago

Someone else reported a possibly similar crash after 3 hours with no plugins, but i cannot replicate it. Can you provide a full backtrace and what format is obs teleport sending audio?

fzwoch commented 1 year ago

I didn't experience the crash myself. The back trace was provided here (or mere the last stack):

https://github.com/fzwoch/obs-teleport/issues/56

Teleport basically just uses what OBS sends to outputs/filters. Same format, just converted from planar to interleaved. Same samples size. It really just grabs it, puts a header in front and off it goes. Same on the receiver, just the same data is handed back.

I suspect that due to a boring hack Teleport could have handed 0 size data to OBS (audio or video). The newest version should not do it if I remember correctly, and it seemed to work fine for OP.

kkartaltepe commented 1 year ago

The user did not attach a full trace so i cannot tell if its the same or not.

It is interesting if it is sending a frame of 0 size. Otherwise this assert exists in shared code, windows also raises asserts so asserting is not platform specific.

kkartaltepe commented 1 year ago

It looks like this user also has a s16le format device which might be the common thing between the two users.

fzwoch commented 1 year ago

Maybe @VictorKruger can revert to the old version for a trace, trying to trigger the error again and make another stack trace (It is basically just typing bt after the crash occurred)

VictorKruger commented 1 year ago

I can certainly do that tonight.

Just to verify, I'm typing "bt" after obs crashes, is that also when I've attached obs do GDB at the start?

On Tue, Oct 11, 2022 at 10:51 AM Florian Zwoch @.***> wrote:

Maybe @VictorKruger https://github.com/VictorKruger can revert to the old version for a trace, trying to trigger the error again and make another stack trace (It is basically just typing bt after the crash occurred)

— Reply to this email directly, view it on GitHub https://github.com/obsproject/obs-studio/issues/7574#issuecomment-1274917636, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQPKLRXGPZDT6NBMJMAH7BTWCWEJ5ANCNFSM6AAAAAARAYRHHQ . You are receiving this because you were mentioned.Message ID: @.***>

fzwoch commented 1 year ago

Yes, attach/run with GDB as before and let it crash. But before exiting GDB you type bt (or maybe someone gives an even more verbose commands that may be helpful)

VictorKruger commented 1 year ago

Just to give an update.

Having 0.6 teleport, and multi-rtmp active when streaming this morning gave me a crash, sadly I didn't have a debugger attached when it happened.

I reverted the teleport plugin to 0.5, removed multi-rtmp and I wasn't able to get obs to crash after 3 hours of testing tonight.

Tomorrow I will ensure that a debugger is attached to obs when I stream. I will have teleport 0.5 and multi-rtmp active since I have been streaming to both youtube and twitch.

VictorKruger commented 1 year ago

Another update. I've haven't had another crash since the last time. I've streamed for over 8 hours on different days. I also either had a cold boot or just left the pc on and did another stream and no crashes.

I may have fixed my issue when reverting back to the old teleport. I wanted to ensure that I didn't have any other oddities to ensure that my crashes were directly involved with teleport and the latest version of OBS, so I performed a purge of OBS, deleting everything except scenes that I created, this included any other obs file(s) within the /usr/ directory and then reinstalled.

I hypothesize that there was a file or files from an older version of OBS that wasn't either deleted or replaced that caused the issue. I've been using OBS for various different needs since around version 26 on linux. This is the first time I've done a full purge and reinstall of the package.

So maybe the best solution to mine and other people's issue on linux is to purge and reinstall?

VictorKruger commented 1 year ago

I finally got a crash tonight. Turns out I needed to have my game pc on windows and then send it over to my linux streaming pc.

https://obsproject.com/logs/zFsCEbr__F8adoyG obsteleportdebug.txt terminal output.txt

so to sum up. linux to linux streams fine, no crashes. win10 to linux the audio drifts in the monitor and then eventually the obs crashes.

--Edit-- Here is the log file for the windows 10 pc https://obsproject.com/logs/I34PQBeCVmo5QIdC

I'm also in process of correcting the issues that were brought up in the analyze.

VictorKruger commented 1 year ago

Just got another crash this time with linux game pc sending video to linux stream pc. I've also noticed a pattern that when OBS crashes and I relaunch and start stream again no audio is being passed to twitch. Once I stop and start the stream again, the audio will work again.

gdb logs from streaming pc gdb terminal output.txt gdblinuxobsteleport.txt

Streaming PC logs: https://obsproject.com/logs/hIpDk3KiinobJBaU https://obsproject.com/logs/g8bbFUdVcfVcW-o7 https://obsproject.com/logs/6C8Kc_RPFFGnw9Fr https://obsproject.com/logs/VpiOgM4yNMcIAojc https://obsproject.com/logs/eQqgdsl1S35pKWPt

Gaming PC log: https://obsproject.com/logs/vNyNcgpE6peatMPK

Hope these logs help out. @fzwoch @kkartaltepe

kkartaltepe commented 1 year ago

Thanks yea those crash logs align with what others have posted. However i cannot reproduce this, so it will probably need someone who can reproduce this and can investigate the fix. If I can figure out how to set up one of my devices to be s16le format i might be able to investigate more.

VictorKruger commented 1 year ago

The device in question that is outputting the s16le format is a Stage Right ix6B from Monoprice.

For your question about altering one of your devices to use s16le, have a look at this forum post from the ubuntu forums. One caveat is that if your device can't support the format, you won't be able to set/force it.

Attached is some more details about the device. Monoprice Stageright ix6B.txt

VictorKruger commented 1 year ago

Hi @kkartaltepe

I went ahead and purchased a new audio interface, a Focusrite Scarlet Solo. I was hoping that this would be a workaround for the time being, however I experienced several crashes today using the new audio interface. The new audio devices uses s32le as a format when recording and playback. I'm thinking that this bug isn't dependent on the audio format used.

Third Crash.txt Second Crash.txt First Crash.txt ScarletSinks.txt ScarletSources.txt teleobsnew.txt teleobsnew2.txt teleobsnew3.txt

fzwoch commented 1 year ago

@VictorKruger can you build OBS by yourself? Currently it seems like there are no debug symbols in your OBS, so the stack how it reaches this point is not quite clear.

Also, when it crashes, perhaps you can traverse the stack a bit down by repeatedly typing up until you reach the point where it actually triggers the assert? At that point type in p size and p *pb. Hopefully it will look like something similar to this then:

(gdb) p size
$6 = 16
(gdb) p *cb
$7 = {data = 0x7fffa0000c80, size = 16, start_pos = 0, end_pos = 16, capacity = 16}

VictorKruger commented 1 year ago

@fzwoch

Attached is another gdb output log. I was unable to perform p size, p cb or p pb, I would get No symbol "cb" in current context. as an error within gdb. When I checked for history, it was empty. I did perform "info registers" on the assert steps so hopefully there will be something in there that is useful.

Also please disregard the large amount of obs browser warnings, from what I can tell, my network's firewall was blocking access to those sites due to ads or tracking. GdbOutput10-27-22.txt

kkartaltepe commented 1 year ago

You left your regular OBS installed, so the debug binaries were not used for libobs where this code resides. You also did not specify the build type or specified RelWithDebInfo, you should generate the cmake config with -DCMAKE_BUILD_TYPE=Debug to remove some optimizations in the code involved and provide better debug information.

VictorKruger commented 1 year ago

Bah, sorry I'll uninstall the packaged version and fully install the build from source obs

On Thu, Oct 27, 2022, 6:44 PM Kurt Kartaltepe @.***> wrote:

You left your regular OBS installed, so the debug binaries were not used for libobs where this code resides.

— Reply to this email directly, view it on GitHub https://github.com/obsproject/obs-studio/issues/7574#issuecomment-1294238558, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQPKLRQCTFQBABJAKZQIVY3WFMHT7ANCNFSM6AAAAAARAYRHHQ . You are receiving this because you were mentioned.Message ID: @.***>

VictorKruger commented 1 year ago

Hi, After some issues getting obs properly compiled for debug and ran, I finally have output that I hope is what's requested of me. gdb.txt

I'll leave this instance running in case you need me do anything else within gdb.

fzwoch commented 1 year ago

Looks like you have enabled audio monitoring for the Teleport source?

Not super familiar with that code, but it looks like a potential race condition to me:

pulseaudio-output.c

        size_t bytesToFill = data->new_data.size;
        if (data->bytesRemaining < bytesToFill)
            bytesToFill = data->bytesRemaining;

In other areas in the code I see:

    pthread_mutex_lock(&data->playback_mutex);
    data->bytesRemaining += nbytes;
    pthread_mutex_unlock(&data->playback_mutex);

In the first code block bytesRemaining is used a couple of times without the playback_mutex being held.

So I can imagine:

if (data->bytesRemaining < bytesToFill) evaluates true
other thread increases bytesRemaining
bytesToFill = data->bytesRemaining sets bytesToFill to a new value higher than anticipated. That value may be higher than what is left in the circlebuffer which was supposedly the maximum to read by setting it with size_t bytesToFill = data->new_data.size.

VictorKruger commented 1 year ago

@fzwoch I was using the monitor feature to listen to the audio coming from the gaming PC to the streaming PC to make sure the audio was properly being sent over. When the crashes became more of an issue, I would also use the monitor as an audio cue that OBS crashed on the streaming PC.

fzwoch commented 1 year ago

Maybe not the most elegant solution, but you could try this patch if it has any impact on the behavior.

--- a/libobs/audio-monitoring/pulse/pulseaudio-output.c
+++ b/libobs/audio-monitoring/pulse/pulseaudio-output.c
@@ -192,8 +192,11 @@ static void do_stream_write(void *param)

        while (data->new_data.size > 0 && data->bytesRemaining > 0) {
                size_t bytesToFill = data->new_data.size;
+
+               pthread_mutex_lock(&data->playback_mutex);
                if (data->bytesRemaining < bytesToFill)
                        bytesToFill = data->bytesRemaining;
+               pthread_mutex_unlock(&data->playback_mutex);

                pulseaudio_lock();
                if (pa_stream_begin_write(data->stream, (void **)&buffer,
@@ -208,7 +211,9 @@ static void do_stream_write(void *param)
                                PA_SEEK_RELATIVE);
                pulseaudio_unlock();

+               pthread_mutex_lock(&data->playback_mutex);
                data->bytesRemaining -= bytesToFill;
+               pthread_mutex_unlock(&data->playback_mutex);
        }

VictorKruger commented 1 year ago

Hi @fzwoch

I updated the pulseaudio-output.c as you listed above and I still received a couple crashes. However instead of circlebuf, this time it's futex-internal. OBS has crashed for me 3 times today and each time it was due to futex, so maybe your patch fixed the circlebuf issue?

attached is the latest futex-internal crash info from gdb. GDB Output 11-1-22.txt

kkartaltepe commented 1 year ago

Thanks for taking a look and testing some changes, looking at this i suspect the problem is actually related to https://github.com/obsproject/obs-studio/blob/master/libobs/audio-monitoring/pulse/pulseaudio-output.c#L258 which seems like iit should be done before the unlock: step. Right now if locks fail to get acquired it will still attempt writes with bad inputs which may be causing the crash. Moving that line up also ensures the whole write is locked so we dont need to add any additional locking to the write function itself.

fzwoch commented 1 year ago

I thought so too. But it was put so obviously out of the lock that I thought it was done on purpose.

The new crash somehow seems involve Qt mouse events.. so I'm not sure how this correlates.. unless the stack gets corrupt.

bunder2015 commented 1 year ago

Hello, I think I have this same problem... I have to leave my obs open all the time because of another issue (my capture card cycles on and off if I'm not actively capturing it - haven't gotten around to updating my kernel to see if that fixes it)

After a seemingly random amount of time, obs crashes on my capture PC with this error.

obs: /var/tmp/portage/media-video/obs-studio-28.1.2-r1/work/obs-studio-28.1.2/libobs/util/circlebuf.h:246: circlebuf_peek_front: Assertion `size <= cb->size' failed.
Aborted

After it crashes, I have to open and close obs several times to get sound working again.

The only plugin I have is NDI, but I have to get my hands on updated ebuilds for ndi-sdk-bin and obs-ndi before I can upgrade to 29 (or to recompile the NDI stuff)

Current build info

[ebuild   R   ~] media-video/obs-studio-28.1.2-r1::gentoo  USE="alsa pipewire pulseaudio ssl truetype v4l wayland -browser -decklink -fdk -jack -lua -nvenc -python (-qt6) -speex -vlc -websocket" LUA_SINGLE_TARGET="luajit" PYTHON_SINGLE_TARGET="python3_10 -python3_8 -python3_9 -python3_11" 0 KiB

When I can recompile everything I'll try to get a full debug build going so I can post a proper crash dump.

Thanks

kkartaltepe commented 1 year ago

Another update on this, we have found a reported underflow in this audo monitor code as well in the same code. Its unclear if its related to me but its certainly not a good thing that its in the same code.

#0  0x00007ffff3ce33bb in getpid () at /usr/lib/x86_64-linux-gnu/libc.so.6
#1  0x00007ffff0b67518 in pa_detect_fork () at /usr/lib/x86_64-linux-gnu/pulseaudio/libpulsecommon-16.0.so
#2  0x00007ffff3e82688 in pa_stream_write_ext_free () at /usr/lib/x86_64-linux-gnu/libpulse.so.0
#3  0x00007ffff3e82ca9 in pa_stream_write () at /usr/lib/x86_64-linux-gnu/libpulse.so.0
#4  0x00007ffff67d0185 in do_stream_write (param=0x5555561a8a40) at /run/build/obs/libobs/audio-monitoring/pulse/pulseaudio-output.c:207
#5  on_audio_playback (param=0x5555561a8a40, source=<optimized out>, audio_data=<optimized out>, muted=<optimized out>)
    at /run/build/obs/libobs/audio-monitoring/pulse/pulseaudio-output.c:258
#6  0x00007ffff6779bdd in source_signal_audio_data (muted=false, in=0x7fffba7dc560, source=0x555556cc4b20)
    at /run/build/obs/libobs/obs-source.c:1374
#7  source_output_audio_data (data=0x7fffba7dc560, source=0x555556cc4b20) at /run/build/obs/libobs/obs-source.c:1562
#8  obs_source_output_audio (audio_in=0x7fffba7dc610, source=0x555556cc4b20) at /run/build/obs/libobs/obs-source.c:4013
#9  obs_source_output_audio (source=0x555556cc4b20, audio_in=audio_in@entry=0x7fffba7dc6b0) at /run/build/obs/libobs/obs-source.c:3978
#10 0x00007fffe400c344 in pulse_stream_read (p=<optimized out>, nbytes=<optimized out>, userdata=0x555555ab12c0)
    at /run/build/obs/plugins/linux-pulseaudio/pulse-input.c:218
#11 0x00007ffff3e63ebe in pstream_memblock_callback () at /usr/lib/x86_64-linux-gnu/libpulse.so.0
#12 0x00007ffff0b99c9f in do_read () at /usr/lib/x86_64-linux-gnu/pulseaudio/libpulsecommon-16.0.so
#13 0x00007ffff0b9a897 in do_pstream_read_write () at /usr/lib/x86_64-linux-gnu/pulseaudio/libpulsecommon-16.0.so
#14 0x00007ffff3e7743b in pa_mainloop_dispatch () at /usr/lib/x86_64-linux-gnu/libpulse.so.0
#15 0x00007ffff3e77aa1 in pa_mainloop_iterate () at /usr/lib/x86_64-linux-gnu/libpulse.so.0
#16 0x00007ffff3e77b58 in pa_mainloop_run () at /usr/lib/x86_64-linux-gnu/libpulse.so.0
#17 0x00007ffff3e87e91 in thread () at /usr/lib/x86_64-linux-gnu/libpulse.so.0
#18 0x00007ffff0ba68db in internal_thread_func () at /usr/lib/x86_64-linux-gnu/pulseaudio/libpulsecommon-16.0.so
#19 0x00007ffff3c8f1da in start_thread () at /usr/lib/x86_64-linux-gnu/libc.so.6
#20 0x00007ffff3d17d84 in clone () at /usr/lib/x86_64-linux-gnu/libc.so.6

$8 = {source = 0x555556cc4b20, stream = 0x555556dbf1a0, device = 0x555556dd8c60 "Private.monitor", attr = {maxlength = 4294967295, 
    tlength = 21600, prebuf = 4294967295, minreq = 4294967295, fragsize = 4294967295}, speakers = SPEAKERS_STEREO, 
  format = PA_SAMPLE_FLOAT32LE, samples_per_sec = 48000, bytes_per_frame = 8, channels = 2 '\002', packets = 451775, 
  frames = 542130000, new_data = {data = 0x7fffa01c68b0, size = 18446737297246488732, start_pos = 484, end_pos = 144000, 
    capacity = 153600}, resampler = 0x555556d3d720, bytesRemaining = 18446737301556138826, ignore = false, playback_mutex = {__data = {
      __lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, 
        __next = 0x0}}, __size = '\000' <repeats 39 times>, __align = 0}}

kkartaltepe commented 1 year ago

I have posted a proposed fix for this issue in https://github.com/obsproject/obs-studio/pull/8303, if you want that PR to land it will need people from this thread to test it and confirm whether or not it fixes the issue. If no one tests it, it will take much longer for it to be merged if ever.

VictorKruger commented 1 year ago

I'll try it out tomorrow and let you know.

On Sun, Feb 19, 2023, 9:30 PM Kurt Kartaltepe @.***> wrote:

I have posted a proposed fix for this issue in #8303 https://github.com/obsproject/obs-studio/pull/8303, if you want that PR to land it will need people from this thread to test it and confirm whether or not it fixes the issue. If no one tests it, it will take much longer for it to be merged if ever.

— Reply to this email directly, view it on GitHub https://github.com/obsproject/obs-studio/issues/7574#issuecomment-1436267579, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQPKLRTCOTCUZ2AAEVQBDSLWYLQL3ANCNFSM6AAAAAARAYRHHQ . You are receiving this because you were mentioned.Message ID: @.***>

RytoEX commented 1 year ago

@VictorKruger @fzwoch Any luck finding time to test the linked PR?

VictorKruger commented 1 year ago

my apologies, real life happened and I forgot to test it. I'm working on getting it setup now and testing it.

On Fri, Feb 24, 2023 at 5:04 PM Ryan Foster @.***> wrote:

@VictorKruger https://github.com/VictorKruger @fzwoch https://github.com/fzwoch Any luck finding time to test the linked PR?

— Reply to this email directly, view it on GitHub https://github.com/obsproject/obs-studio/issues/7574#issuecomment-1444632569, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQPKLRQEPR543Y5TXKXFLTTWZE5ARANCNFSM6AAAAAARAYRHHQ . You are receiving this because you were mentioned.Message ID: @.***>

obsproject / obs-studio