AV1 YouTube bitrate drop until stream restart

robdiesalot commented 5 months ago

Operating System Info

Windows 11

Other OS

No response

OBS Studio Version

30.1.0

OBS Studio Version (Other)

No response

OBS Studio Log URL

https://obsproject.com/logs/VJcmhy0uy4lERRpI

OBS Studio Crash Log URL

No response

Expected Behavior

A stable AV1 stream to YouTube with a fairly beefy dedicated streaming PC.

Current Behavior

Nvidia AV1 stream to YouTube randomly drops bitrate to ~160kbps and stays there until stream is restarted.

Steps to Reproduce

1. 2. 3. ...

Anything else we should know?

I have been trying to stream with Nvidia AV1 for about a year now (since AV1 OBS release) and tried every new update. I never have similar issues with other encoders, so I dont think it has anything to do with my ISP. Im afraid the log doesnt show anything except : 13:31:45.549: Output 'adv_stream': Number of dropped frames due to insufficient bandwidth/connection stalls: 6283 (1.0%) But let me repeat, my internet runs just fine and this only happens with AV1.

During my stream (usually 16000 kbps), the bitrate drops to ~160kbps and stays there until I restart the stream. Nothing seems to trigger this unfortunately and it happens at random time intervals (sometimes runs fine for days, other times drops every 10 minutes)

This issue seems to be random, but I think it happens more during weekends and busy hours. (So possibly YT servers?)

I have been searching for other users with similar problems and I found a lot of them, but noone seems to have a clue about why this could be happening: https://obsproject.com/forum/threads/av1-live-streaming-on-youtube-not-working-please-help.168513/

RytoEX commented 5 months ago

Number of memory leaks: 458

Please reproduce this without any third-party plugins.

Please also test with any other codec. Dropped frames are purely related to network conditions and they are handled separately from the encoding process, so I cannot imagine this has anything to do with the codec choice.

Please also test with HLS. Provide logs for each different test.

I have been searching for other users with similar problems and I found a lot of them, but noone seems to have a clue about why this could be happening: https://obsproject.com/forum/threads/av1-live-streaming-on-youtube-not-working-please-help.168513/

The first user's case sounds like typical dropped frames due to fluctuating network conditions.

That is usually enough to fix, but still makes AV1 streaming unbearable. Ive been trying it since OBS 29 through all the patches, but no changes.

This indicates that this isn't a new issue for 30.1.

robdiesalot commented 5 months ago

Please also test with any other codec. Dropped frames are purely related to network conditions and they are handled separately from the encoding process, so I cannot imagine this has anything to do with the codec choice.

Thats the thing, it never happens with other codecs and I have thousands of hours streamed. With 3rd party plugins and without them. Could it just be AV1 not being able to reconnect after a brief ISP drop maybe?

RytoEX commented 5 months ago

Please also test with any other codec. Dropped frames are purely related to network conditions and they are handled separately from the encoding process, so I cannot imagine this has anything to do with the codec choice.

Thats the thing, it never happens with other codecs and I have thousands of hours streamed. With 3rd party plugins and without them. Could it just be AV1 not being able to reconnect after a brief ISP drop maybe?

AV1 is a codec, not a streaming protocol. The codec isn't responsible for reconnection. There could be two separate issues:

Dropped frames occurring due to network conditions.
The video encoder dying.

robdiesalot commented 5 months ago

Dropped frames occurring due to network conditions.

The video encoder dying. Yes, thats what Im thinking as a possibility.

Frames drop because a momentary network drop

Encoder dies instead of catching up

Even when my internet dies for a few seconds, h264 codecs keep running and after a while bitrate goes back up and stabilizes. With AV1 stream needs to be restarted.

ArktixOfficial commented 5 months ago

Hello. Just wanted to add that I'm having the same issue in March 2024 with RTX 4070Ti on the most current driver. Trying to use AV1 to stream to YouTube and the stream bitrate drops randomly and requires a restart in OBS. Very frustrating. I will be reverting back to HEVC or H.264 until this apparent AV1 issue gets resolved. From what I'm seeing, I think it's either an issue with NVIDIA hardware/firmware (more likely firmware), OBS implementation of AV1, or YouTube's ingest of AV1. Difficult to pin down which one, but it's clear steaming AV1 to YouTube with OBS has issues.

RytoEX commented 5 months ago

Hello. Just wanted to add that I'm having the same issue in March 2024 with RTX 4070Ti on the most current driver. Trying to use AV1 to stream to YouTube and the stream bitrate drops randomly and requires a restart in OBS. Very frustrating. I will be reverting back to HEVC or H.264 until this apparent AV1 issue gets resolved. From what I'm seeing, I think it's either an issue with NVIDIA hardware/firmware (more likely firmware), OBS implementation of AV1, or YouTube's ingest of AV1. Difficult to pin down which one, but it's clear steaming AV1 to YouTube with OBS has issues.

Please provide a log file from when this occurs, preferably without any third-party plugins.

robdiesalot commented 5 months ago

So I have been using HEVC for the past few days and it runs PERFECTLY with the exact same setup, plugins and OBS version (also 4070Ti). So maybe it really is an Nvidia issue. HEVC resumes streaming just fine while AV1 dies and requires restart.

RytoEX commented 5 months ago

So I have been using HEVC for the past few days and it runs PERFECTLY with the exact same setup, plugins and OBS version (also 4070Ti). So maybe it really is an Nvidia issue. HEVC resumes streaming just fine while AV1 dies and requires restart.

Please provide a new log file when you retest.

LucasSnatiago commented 5 months ago

So I'm using a AMD RX7600 with AV1 on a Linux machine (Arch Linux kernel: 6.8.1-arch1-1) and having the same problem than you guys. So, it can be a think not exclusive to NVIDIA cards. It happens for me after one hour of streaming in average, then it keeps happening for every five minutes needing to open OBS Studio again. It is very annoying. I don't use any third-party plugins and I still have all the issues commented above.

RytoEX commented 5 months ago

So I'm using a AMD RX7600 with AV1 on a Linux machine (Arch Linux kernel: 6.8.1-arch1-1) and having the same problem than you guys. So, it can be a think not exclusive to NVIDIA cards. It happens for me after one hour of streaming in average, then it keeps happening for every five minutes needing to open OBS Studio again. It is very annoying. I don't use any third-party plugins and I still have all the issues commented above.

Please provide a log file.

ArktixOfficial commented 5 months ago

So I'm using a AMD RX7600 with AV1 on a Linux machine (Arch Linux kernel: 6.8.1-arch1-1) and having the same problem than you guys. So, it can be a think not exclusive to NVIDIA cards. It happens for me after one hour of streaming in average, then it keeps happening for every five minutes needing to open OBS Studio again. It is very annoying. I don't use any third-party plugins and I still have all the issues commented above.

Good to know. Thanks for sharing. This helps narrow it down a bit. I think the fact that your having this issue on an AMD card pretty much eliminates the possibility of it being something to do with the GPU hardware or firmware. So, it must be something with OBS, YouTube, or both.

ArktixOfficial commented 5 months ago

@RytoEX I just completed a 3 hour test stream session with AV1 encoding to YouTube to collect a good log file. I uploaded via the OBS upload prompt. Here is the URL for the log (also attached at the end of this post): https://obsproject.com/logs/EfrhJz5VpyBsHJ0A

I did leave 2 plugins installed because these are critical for my workflow and wanted to test with them running to see if they are causing the issue. The 2 plugins are the Multi RTMP plugin and the Stream Deck plugin. I also often use Source Record, but I uninstalled it for this test. If you absolutely need a log with no 3rd party plugins installed, let me know and I can try to get another test in.

Here are other parameters of the test so people can easily reference them:

Internet upload speed: 22-25 Mbps over wired ethernet CPU: Intel 13900K GPU: NVIDIA 4070 Ti (This is a dedicated streaming PC. The game is run on a separate PC/console and I use a capture card)

YouTube Stream (set up through OBS default streaming output)

Service: YouTube - RTMPS Server: YouTube Primary Ingest Server Connection: Use Stream Key (my account is not connected, but I have tested it both ways and seen the same issue) Base Canvas Res: 3840x2160 Output Res: 2560x1440 Downscale filter: Lanczos Audio Encoder: FFmpeg AAC Video Encoder: NVIDIA NVENC AV1 Rate Control: CBR Bitrate: 11500 Kbps Keyframe Interval: 2s Preset: P6: Slower Tuning: High Quality Multipass Mode: Two Passes (Quarter Res) Profile: main Look Ahead: Off Psycho Visual Tuning: On Max B-Frames: 2

Twitch Stream (set up through Multi RTMP plugin)

Res: 1920x1080 Audio Encoder: Get from OBS Video Encoder: NVIDIA NVENC H.264 (FFmpeg) (ffmpeg_nvenc) Rate Control: CBR Bitrate: 6000 Kbps Keyframe Interval 2s Preset: P6: Slower Tuning: High Quality Multipass Mode: Two Passes (Quarter Res) Profile: High Psycho Visual Tuning: On Max B-Frames: 2

2024-03-25 12-20-47.txt

Any insights you might be able to share would be much appreciated. Other things I have considered testing, but have not yet: -Change YouTube server to something other than the primary ingest server -Change service to YouTube - HLS -Send the YouTube Stream through Multi RTMP and Twitch through default OBS

RytoEX commented 5 months ago

@ArktixOfficial Please provide a log of when the issue occurs. Ideally, yes, one without third-party plugins would ensure that only our code is running. If you cannot get a log without third-party plugins, then as few as possible will have to do, but zero really is best.

ArktixOfficial commented 5 months ago

@RytoEX Sorry if I didn’t make this clear, but the log file I provided is from a 3 hour test stream where the issue did occur. It occurred towards the end of the session and I stopped and restarted the stream in OBS to revive the YouTube stream. Let me know if I can provide any additional info.

I will try to run a test without plugins installed, but I have limited time where I can lock up all my upload bandwidth for a test during the next week or so. Not sure when I’ll be able to do it.

LucasSnatiago commented 5 months ago

I made an extensive test just like @ArktixOfficial. I have 1Gbps down 100 Mbps up internet speed, I stream at 25Mbps. I used during all the test the default network settings, with dynamic change stream bit rate enabled. I have IPV6 and IPV4 working perfectly here.

All tests were made recording to YouTube:

Recording with H265 1 hour and 20 minutes, no problems, just a little network drop in the middle, but recovered successfully and worked to the end of the stream: 1st test
Recording with AV1, live crashed twice, so I needed to stop streaming and start it again twice: 2nd test
I noticed that I had a third-party plugin installed, so I uninstalled it and tried again (and the crash happened again the same way as before, and all tests from now on were made without this plugin): 3rd test
I tried a different YouTube server (legacy one) crashed again: 4th test
I changed back to H265 and tried one last time without using AV1, and it worked: 5th test

After hours of tests I found one important fact that happens very frequently. When using H265 when I have an internet drop it can recover from it and keeps running the stream, but when I use AV1, if any network inconsistency happen, the streaming bit rate spirals down to 166/167kbps and keeps there until I stop and start streaming again. It simple does not recover from this drop. I found a simple way to force a network inconsistency, you just need to open any internet speed tester and run it [^1][^2]. When it goes to the upload part and goes up to the maximum upload speed:

H265: 25Mbps -> 20Mbps -> 25Mbps (It goes from 25Mbps to 20Mbps during the test and goes back to 25Mbps when it finishs).
AV1: 25Mbps -> 20Mbps -> 166/167Kbps (It goes from 25Mbps to 20Mbps then it suddenly crashes?!? to 166 or 167Kbps and keeps there forever).

[^1]: For some reason, when I test my internet speed against my ISP it does not drop my internet speed at all. However, when I test it against any other ISP it crashes my OBS. So keep it in mind in case it happens with you as well. Just try a different ISP and see if it changes the result. [^2]: Anyone in my house can do this test and crash my obs. It doesn't need to be my computer testing.

LucasSnatiago commented 5 months ago

After some consideration, I've noticed that the bit rate drops to approximately 160kbps, coincidentally matching the default audio bit rate. This issue might stem from the AV1 video encoder crashing, resulting in only the audio being transmitted to YouTube's servers. What do you think @RytoEX?

RytoEX commented 5 months ago

After some consideration, I've noticed that the bit rate drops to approximately 160kbps, coincidentally matching the default audio bit rate. This issue might stem from the AV1 video encoder crashing, resulting in only the audio being transmitted to YouTube's servers. What do you think @RytoEX?

Sounds plausible. We have a vague idea of what could be happening, but we'll need to investigate further.

RytoEX commented 5 months ago

For those affected, please test the arfifacts, when available, from https://github.com/obsproject/obs-studio/pull/10436 if you are able to do so.

ArktixOfficial commented 5 months ago

After some consideration, I've noticed that the bit rate drops to approximately 160kbps, coincidentally matching the default audio bit rate. This issue might stem from the AV1 video encoder crashing, resulting in only the audio being transmitted to YouTube's servers. What do you think @RytoEX?

I have noticed the same. A bit of additional info that may be helpful: While streaming, my viewers have noticed that the YouTube stream often goes down right around intense combat scenes and/or when I’m laughing or otherwise being loud in my mic. Not all crashes occur during these times, but it seems increased audio data may have some influence on triggering the stream to crash. However, my audio bite rate is set to 320Kbps in OBS, but YouTube still tells me that it’s getting around 160Kbps. OBS is still showing that it’s pushing the full 11,500Kbps I set.

It does seem like an odd coincidence that YouTube reports it’s getting a bit rate roughly equivalent to the default audio bit rate during these stream crash instances. Perhaps when increased audio data is being received by YouTube it hooks those larger audio packets as the video packets and gets confused about the bite rate it’s receiving? I’m not a dev, so just my speculation. Apologies if I’m using terminology incorrectly.

ArktixOfficial commented 5 months ago

@LucasSnatiago Thanks for your additional tests and info! I agree it seems to be that the video encoder dies and only the audio is being sent to YouTube. I’m not sure how to reconcile the fact that my output audio bitrate is set to 320Kbps, but YouTube tells me it’s receiving roughly 160Kbps during the crash periods. Perhaps it’s not able to accept 320Kbps or maybe it’s compressing the audio before reporting that figure, not sure.

Good thinking running a speed test to force a network inconsistency! I will try that next time I’m able to run a test.

LucasSnatiago commented 5 months ago

@derrod it seems that there is an error during the generation of the artifacts for Linux. It didn't generate any artifacts for Ubuntu.

derrod commented 5 months ago

@derrod it seems that there is an error during the generation of the artifacts for Linux. It didn't generate any artifacts for Ubuntu.

Should be fine now: https://github.com/obsproject/obs-studio/actions/runs/8440309705?pr=10436

LucasSnatiago commented 5 months ago

Fixed!!! I conducted three additional tests to further investigate the issue:

H265 Test: Ran for 2 hours and 30 minutes without any issues. View 1st test
AV1 Test (from @derrod build): Ran for 2 hours without any problems. View 2nd test
OBS Default Test: Ran for only 2 minutes without the derrod patch. View 3rd test

For each test, I disconnected my internet cable twice, waited for 5 seconds each time, and then reconnected it. Both AV1 and H265 were able to recover from the network failure and continue running for hours without crashing. However, during the 2-minute test using default OBS settings, it crashed twice.

@ArktixOfficial, could you please conduct similar tests? A 2-minute test for each scenario should suffice. Simply follow the steps outlined above.

ArktixOfficial commented 5 months ago

@LucasSnatiago Thanks for doing those additional tests and sharing the results! I’m happy to do some tests as well. I should be able to within the next few days. Do you know if there are their instructions I can reference for how to install/utilize the new build? I’m not familiar with artifacts or how to use them. Sorry, just not very knowledgeable in this area and I don’t want to mess up my OBS while I need to use it for production. If there’s a simple way for me to test and then revert back to my current version of OBS, I’m happy to do some tests.

LucasSnatiago commented 5 months ago

@ArktixOfficial I can help you to do your tests, first what is your operating system? So I can explain all the steps here.

ArktixOfficial commented 5 months ago

@LucasSnatiago Thanks! I’m on Windows 11.

obsproject / obs-studio