search

obsproject / obs-studio

OBS Studio - Free and open source software for live streaming and screen recording
https://obsproject.com
GNU General Public License v2.0
57.76k stars 7.74k forks source link

Crash when livestreaming using SRT #10256

Closed Abeja27 closed 4 months ago

Abeja27 commented 5 months ago

Operating System Info

Windows 10

Other OS

No response

OBS Studio Version

30.1.0-beta1

OBS Studio Version (Other)

No response

OBS Studio Log URL

https://obsproject.com/logs/FOhy1iAx3mvWB7Gr

OBS Studio Crash Log URL

https://obsproject.com/logs/QCCBqe12ciKu0RWR

Expected Behavior

Being able to live stream using SRT without any issues.

Current Behavior

After streaming smoothly for a while, the server stops receiving packets from OBS. At this point, OBS sometimes crashes directly, while other times it appears not to, but as soon as you click "Stop Streaming," it freezes completely, requiring manual process termination.

Steps to Reproduce

  1. Initiate a live stream using the SRT protocol. I haven't been able to find a configuration where this issue doesn't occur.
  2. Wait until the issue occurs. The time it takes to fail has varied for me, ranging from 1 minute to 1 hour.

Anything else we should know?

This problem seems to be related to #9438. That issue was still happening to me until this version when finally no packets are lost at all. However that was replaced for the crash/freezing that I'm reporting now.

The previous issue was closed at the time, and I didn't have time to continue investigating. The only discovery I made is that the problem worsens significantly when setting the SRT option "sendbuf" to a high value. With small values, there were almost no lost packets, but it wasn't a viable workaround as the effective bitrate decreased due to the small buffer size. In this version, I'm unsure if that option has any impact anymore.

It would be great to know the exact commit where the behavior changed. Do you have any idea which one could I check? I wouldn't like to test one by one.

I would be great If you could reproduce it but that wasn't the case with #9438. Maybe it could be related to some specific hardware or configuration, but after testing different configurations in 3 machines It could find the issue. Ask me for any further information you may need.

Thank you for your time.

pkviet commented 5 months ago

i need clear repro steps as i don't observe this at all; on the three machines you observe the issue, is it on the same network ? if it is, i'd suspect some router settings ?

RealIndrit commented 5 months ago

"After streaming smoothly for a while, the server stops receiving packets from OBS. At this point, OBS sometimes crashes directly, while other times it appears not to, but as soon as you click "Stop Streaming," it freezes completely, requiring manual process termination."

Is there a possibility to get a memory dump where the process is frozen using .dmp file generated by windows? Could be helpful in troubleshooting the freezing of the process at least.

Abeja27 commented 5 months ago

I just realized that it does work correctly if the SRT server is on localhost, no packet loss and no crash/freezing . I could swear I had tested this before, but I must have done something wrong. However I think it can't be a problem in the server side because if you capture the packets using Wireshark in the OBS machine, you see that those packets are missing there too.

i need clear repro steps as i don't observe this at all; on the three machines you observe the issue, is it on the same network ? if it is, i'd suspect some router settings ?

On most tests I've done, the server was on my local network. I'm not sure what could cause this. I have tried to directly connect the Ethernet cable from one machine to the other and I was able to repro the issue, so router/switch is not the problem here.

To make it easier for you to replicate the issue, here's a more specific case to reproduce:

  1. Create an SRT server with FFMPEG in a machine of your local network, in this example 192.168.1.103: ./ffmpeg -i 'srt://:40009?mode=listener' -f null /dev/null
  2. Connect to the server with OBS: srt://192.168.1.103:40009?latency=10000&sndbuf=500000000

if I do that previous to this beta In this beta OBS crashes or freezes after a while. Previous to this beta I get the following in the SRT server: image

However, if I use FFMPEG instead of OBS, It usually works fine. However today, I got corrupted frames when decoding in the server. First time I see this. But as you can see, there were no packet loss as in OBS: image

I am confused after this. Maybe is it a ffmpeg bug then? drivers? I've updated graphic card driver and no luck.

For sending video using o OBS I used the following command: ./ffmpeg -f lavfi -re -i smptebars=duration=3000:size=1920x1080:rate=60 -pix_fmt yuv420p -c:v h264_nvenc -b:v 20M -minrate 20M -maxrate 20M -rc cbr -bufsize 200M -g 30 -keyint_min 30 -profile:v high -preset slow -f mpegts "srt://192.168.1.100:40009?latency=10000&sndbuf=500000000"

To force this issue to happen, it helps me a lot to increase CPU usage by running a CPU stress test. You can also try to change the sndbuf parameter value.

Is there a possibility to get a memory dump where the process is frozen using .dmp file generated by windows? Could be helpful in troubleshooting the freezing of the process at least.

I have no idea about how to do that, but if you tell me what to do I'll do it.

laurfb commented 5 months ago

Hi. Same here. OBS v30.1 Beta2 halt/output staled after 30-40 min. Twice, with no apparent reason. Also in my case we use a medium to high value for sndbuf (28000000), 650ms latency with 10Mbps bitrate. Need to manual close it. Stop Streaming / Start Streaming simply not working. Revert to v30.0 stable. No trouble since them. Only found this in output log, don't know if it helps. sshot-218

pkviet commented 5 months ago

We've updated libsrt in the beta to 1.5.3; as well as the avformat lib (from FFmpeg) which is used to mux to mpeg-ts format. Here's the git history for the srt changes in the code: https://github.com/obsproject/obs-studio/commits/master/plugins/obs-ffmpeg/obs-ffmpeg-srt.h I made a change for rendezvous mode but nothing jumps out which might explain the issue.

I'll try to make more tests to trigger the bug.

pkviet commented 5 months ago

ps: this PR https://github.com/obsproject/obs-studio/pull/9027 displays more stats and could help to pinpoint the issue. You'll see all retransmitted packets in a graph. There are artefacts which you can download. They're based on obs 30 though and not the beta. I'll update the PR later.

pps: the srt code is almost all borrowed from FFmpeg; so it's very odd. But I'll double check if I spot something.

RealIndrit commented 5 months ago

@Abeja27

Now idk how much this would help, other than see exact location and memory state of OBS when it crashes. I personally do it directly through the Task Manager, it should be an option when right clicking the process in the active process list. Used preferably on a OBS Release build with debug info active.

Steps to acquire dmp file:

  1. Press Ctrl+Alt+Delete on the keyboard.
  2. Select Task Manager .
  3. If you are using: Windows 7, go to the Processes tab. Windows 11, the Processes tab will be opened by default. Proceed to the next step. Windows 8, 8.1, 10 or Windows Server 2008, click More details.
  4. Right-click the process for which you need to create a dump file. Select Create Dump File.

To actually get something useful out of it I recommend using WinDbg to analyze it and see what's going on. by looking at the Call Stacks and memory fields, at the time of the crash.

thelegendtubaguy commented 5 months ago

Not sure if my issue is exactly the same... I get 1 or just a few frames of video output. Output seems to continue to send, but I don't get any further video frames or more audio. Video preview continues to work. When I click "stop streaming" OBS hangs and I have to kill it. It does this every time so I should be able to repro easily. Will try to get a dump.

Edit The memory dump of the main OBS process is almost 7 gigs... idk if that would actually help anyone but let me know if you think so. I tried this a few more times and find that I randomly do get beyond that initial few frames of audio/video and it sends for minutes... but eventually hangs and stops sending.

pkviet commented 5 months ago

Not sure if my issue is exactly the same... I get 1 or just a few frames of video output. Output seems to continue to send, but I don't get any further video frames or more audio. Video preview continues to work. When I click "stop streaming" OBS hangs and I have to kill it. It does this every time so I should be able to repro easily. Will try to get a dump.

Edit The memory dump of the main OBS process is almost 7 gigs... idk if that would actually help anyone but let me know if you think so. I tried this a few more times and find that I randomly do get beyond that initial few frames of audio/video and it sends for minutes... but eventually hangs and stops sending.

if you go back to obs 30.0, there's no issue ? if you confirm, my suspicion will be on libsrt which we updated. I'll provide then a test build with previous version of libsrt but with current beta. For the dump, what's useful for me is the stack trace; you can open the dump with windbg and then enter !analyze -v

thelegendtubaguy commented 5 months ago

That is correct. I go back to 30 and there's no issue at all. I've changed nothing between versions related to my SRT config.

I'll get that stack trace today! Thanks!

RytoEX commented 5 months ago

If you are having this issue, please provide the full configuration of your SRT parameters (you can omit the domain/IP and port information). Please also provide the full information for your SRT server/receiver (software, version, parameter configuration), whether OBS and the server/receiver are on the same network or same PC (or different network/PC), and the relevant OBS log for the problem session.

If you want to bisect the exact point at which dependencies changed, here are relevant commits (which point to different dependency versions) which should have CI artifacts available:

thelegendtubaguy commented 5 months ago

I use ffmpeg in listener mode to output the OBS stream to a decklink interface. Listener command looks like this:

ffmpeg -re -i "srt://0.0.0.0:1935?mode=listener" -async 3 -fps_mode cfr -bufsize 1024M -f decklink -pix_fmt uyvy422 -b:v 15000k -threads 12 "DeckLink Mini Monitor HD"

The server field in OBS looks like this:

srt://192.168.1.5:1935?mode=caller

This is on a local wired 10 gig network, two different computers.

Log file is here: https://obsproject.com/tools/analyzer?log_url=https%3A%2F%2Fobsproject.com%2Flogs%2FGOstBEY5U66LqsOS#logURL

Sorry for the amount of sources 😅 Please keep in mind that I can go back to OBS 30 and run without issues.

thelegendtubaguy commented 5 months ago

Not sure if my issue is exactly the same... I get 1 or just a few frames of video output. Output seems to continue to send, but I don't get any further video frames or more audio. Video preview continues to work. When I click "stop streaming" OBS hangs and I have to kill it. It does this every time so I should be able to repro easily. Will try to get a dump. Edit The memory dump of the main OBS process is almost 7 gigs... idk if that would actually help anyone but let me know if you think so. I tried this a few more times and find that I randomly do get beyond that initial few frames of audio/video and it sends for minutes... but eventually hangs and stops sending.

if you go back to obs 30.0, there's no issue ? if you confirm, my suspicion will be on libsrt which we updated. I'll provide then a test build with previous version of libsrt but with current beta. For the dump, what's useful for me is the stack trace; you can open the dump with windbg and then enter !analyze -v

This is what I got: https://gist.github.com/thelegendtubaguy/3266ef0b4920345b10a93ebe23a4de06

pkviet commented 5 months ago

Rytoex has a stack trace which seems to point to a libsrt bug introduced in 1.5.3 which has later been fixed See: https://github.com/Haivision/srt/issues/2841 & https://github.com/Haivision/srt/pull/2834

@thelegendtubaguy @Abeja27 @laurfb Could you test by replacing srt.dll in bin folder with these? libsrt_master.zip (srt was built with openssl instead of mbedtls in this build, hence the second dll, just because it's faster for me)

laurfb commented 5 months ago

Rytoex has a stack trace which seems to point to a libsrt bug introduced in 1.5.3 which has later been fixed See: Haivision/srt#2841 & Haivision/srt#2834

@thelegendtubaguy @Abeja27 @laurfb Could you test by replacing srt.dll in bin folder with these? libsrt_master.zip (srt was built with openssl instead of mbedtls in this build, hence the second dll, just because it's faster for me)

Yes, been running for 10 min. already. Thx! regards, Laur

laurfb commented 5 months ago

Hi again. For me, seems to be Ok, so far. Almost 2 hours without any glitch. OBS v30.1.0-beta2 with provided srt lib (v1.5.2) Thx! regards, Laur

thelegendtubaguy commented 5 months ago

That fixes it for me! I can't get it to exhibit the same behavior as before. Either in lots of short term tests or streaming for tens of minutes, where previously it could not.

Abeja27 commented 5 months ago

Fixed for me too. I couldn't reproduce it again after 2 hours of testing. @RealIndrit I'm sorry, you asked me the dump files but I didn't have any time these days. Too late now I guess. Thank you guys your fast support.

pkviet commented 5 months ago

Thanks for the feedbacks.

RytoEX commented 5 months ago

Rytoex has a stack trace

To clarify, I only got this to freeze once during some brief testing. I haven't had OBS outright crash from it. If there's a separate crash scenario, I haven't found it yet.