OBS freezes while recording after changing the monitor's refresh rate

[nstepien]

Platform

Operating system and version: win10 20H2 OBS Studio version: 26.0.2

Expected Behavior

Changing the monitor's refresh rate shouldn't lock up OBS when recording or with the replay buffer active, maybe streaming as well but I haven't tested this.

Current Behavior

I've been playing Dark Souls 3 with the replay buffer on, my desktop refresh rate is 144Hz, but the game forces the monitor refresh rate to 60Hz. When I alt+tab out of the game the monitor refresh rate changes, and OBS can become unresponsive and I have to terminate its process in the task manager. It doesn't always lock up on the first alt+tab, it can take a few tries, but it's not rare. If I set my desktop refresh rate to 60Hz then it all works fine.

Log:

Click to view log

``` 11:48:55.547: CPU Name: AMD Ryzen Threadripper 3970X 32-Core Processor 11:48:55.547: CPU Speed: 3693MHz 11:48:55.547: Physical Cores: 32, Logical Cores: 64 11:48:55.547: Physical Memory: 32661MB Total, 27316MB Free 11:48:55.547: Windows Version: 10.0 Build 19042 (release: 2009; revision: 630; 64-bit) 11:48:55.547: Running as administrator: true 11:48:55.547: Aero is Enabled (Aero is always on for windows 8 and above) 11:48:55.547: Windows 10 Gaming Features: 11:48:55.547: Game Bar: Off 11:48:55.548: Game DVR: Off 11:48:55.548: Game DVR Background Recording: Off 11:48:55.548: Hardware GPU Scheduler: On 11:48:55.549: Sec. Software Status: 11:48:55.549: Microsoft Defender Antivirus: enabled (AV) 11:48:55.550: Windows Firewall: enabled (FW) 11:48:55.550: Current Date/Time: 2020-11-19, 11:48:55 11:48:55.550: Browser Hardware Acceleration: true 11:48:55.550: Portable mode: false 11:48:55.741: OBS 26.0.2 (64-bit, windows) 11:48:55.741: --------------------------------- 11:48:55.741: --------------------------------- 11:48:55.741: audio settings reset: 11:48:55.741: samples per sec: 48000 11:48:55.741: speakers: 2 11:48:55.743: --------------------------------- 11:48:55.743: Initializing D3D11... 11:48:55.743: Available Video Adapters: 11:48:55.744: Adapter 0: NVIDIA GeForce GTX 1080 11:48:55.744: Dedicated VRAM: 4176478208 11:48:55.744: Shared VRAM: 4239173632 11:48:55.744: PCI ID: 10de:1b80 11:48:55.744: Driver Version: 27.21.14.5730 11:48:55.744: output 0: pos={0, 0}, size={2560, 1440}, attached=true, refresh=144, name=XB270HU 11:48:55.746: Loading up D3D11 on adapter NVIDIA GeForce GTX 1080 (0) 11:48:55.797: D3D11 loaded successfully, feature level used: b000 11:48:55.797: DXGI increase maximum frame latency success 11:48:55.798: D3D11 GPU priority setup success 11:48:56.113: --------------------------------- 11:48:56.113: video settings reset: 11:48:56.113: base resolution: 2560x1440 11:48:56.113: output resolution: 2560x1440 11:48:56.113: downscale filter: Bicubic 11:48:56.113: fps: 60/1 11:48:56.113: format: NV12 11:48:56.113: YUV mode: 709/Partial 11:48:56.114: NV12 texture support enabled 11:48:56.115: Audio monitoring device: 11:48:56.115: name: Default 11:48:56.115: id: default 11:48:56.115: --------------------------------- 11:48:56.118: [CoreAudio encoder]: CoreAudio AAC encoder not installed on the system or couldn't be loaded 11:48:56.118: Failed to load 'en-US' text for module: 'decklink-ouput-ui.dll' 11:48:56.139: [AMF] Unable to load 'amfrt64.dll', error code 126. 11:48:56.141: [AMF] AMF Test failed due to one or more errors. 11:48:56.141: Failed to initialize module 'enc-amf.dll' 11:48:56.154: [obs-browser]: Version 2.8.7 11:48:56.156: NVENC supported 11:48:56.230: Couldn't find VLC installation, VLC video source disabled 11:48:56.235: A DeckLink iterator could not be created. The DeckLink drivers may not be installed 11:48:56.235: No blackmagic support 11:48:56.238: --------------------------------- 11:48:56.239: Loaded Modules: 11:48:56.239: win-wasapi.dll 11:48:56.239: win-mf.dll 11:48:56.239: win-dshow.dll 11:48:56.239: win-decklink.dll 11:48:56.239: win-capture.dll 11:48:56.239: vlc-video.dll 11:48:56.239: text-freetype2.dll 11:48:56.239: rtmp-services.dll 11:48:56.239: obs-x264.dll 11:48:56.239: obs-vst.dll 11:48:56.239: obs-transitions.dll 11:48:56.239: obs-text.dll 11:48:56.239: obs-qsv11.dll 11:48:56.239: obs-outputs.dll 11:48:56.239: obs-filters.dll 11:48:56.239: obs-ffmpeg.dll 11:48:56.239: obs-browser.dll 11:48:56.239: image-source.dll 11:48:56.239: frontend-tools.dll 11:48:56.239: enc-amf.dll 11:48:56.239: decklink-ouput-ui.dll 11:48:56.239: coreaudio-encoder.dll 11:48:56.239: --------------------------------- 11:48:56.239: ==== Startup complete =============================================== 11:48:56.242: All scene data cleared 11:48:56.242: ------------------------------------------------ 11:48:56.313: WASAPI: Device 'Speakers (DX7s)' [192000 Hz] initialized 11:48:56.313: [Loaded global audio device]: 'Desktop Audio' 11:48:56.336: WASAPI: Device 'Microphone (Samson GoMic)' [48000 Hz] initialized 11:48:56.337: [Loaded global audio device]: 'Mic/Aux' 11:48:56.337: - filter: 'Noise Suppression' (noise_suppress_filter_v2) 11:48:56.337: [duplicator-monitor-capture: 'Display Capture'] update settings: 11:48:56.337: display: 1 (2560x1440) 11:48:56.337: cursor: true 11:48:56.393: adding 64 milliseconds of audio buffering, total audio buffering is now 64 milliseconds (source: Mic/Aux) 11:48:56.393: 11:48:56.689: Switched to scene 'Display capture' 11:48:56.691: ------------------------------------------------ 11:48:56.691: Loaded scenes: 11:48:56.691: - scene 'Game capture': 11:48:56.691: - source: 'Game Capture' (game_capture) 11:48:56.691: - scene 'Display capture': 11:48:56.691: - source: 'Display Capture' (monitor_capture) 11:48:56.691: ------------------------------------------------ 11:49:00.017: [jim-nvenc: 'recording_h264'] settings: 11:49:00.017: rate_control: CQP 11:49:00.017: bitrate: 0 11:49:00.017: cqp: 10 11:49:00.017: keyint: 250 11:49:00.017: preset: hq 11:49:00.017: profile: high 11:49:00.017: width: 2560 11:49:00.017: height: 1440 11:49:00.017: 2-pass: false 11:49:00.017: b-frames: 2 11:49:00.017: lookahead: false 11:49:00.017: psycho_aq: true 11:49:00.017: 11:49:00.028: --------------------------------- 11:49:00.028: [FFmpeg aac encoder: 'desktop'] bitrate: 320, channels: 2, channel_layout: 3 11:49:00.028: 11:49:00.029: --------------------------------- 11:49:00.029: [FFmpeg aac encoder: 'mic'] bitrate: 320, channels: 2, channel_layout: 3 11:49:00.029: 11:49:00.030: --------------------------------- 11:49:00.030: [FFmpeg aac encoder: 'desktop+mic'] bitrate: 320, channels: 2, channel_layout: 3 11:49:00.030: 11:49:00.038: ==== Recording Start =============================================== 11:49:00.038: [ffmpeg muxer: 'adv_file_output'] Writing file 'D:/obs/2020-11-19 11-48-59.mkv'... 11:50:33.617: [jim-nvenc] nvenc_encode_tex: nvEncEncodePicture failed: 4 (NV_ENC_ERR_INVALID_DEVICE) 11:50:33.617: Error encoding with encoder 'recording_h264' 11:50:33.624: [ffmpeg muxer: 'adv_file_output'] Output of file 'D:/obs/2020-11-19 11-48-59.mkv' stopped 11:50:33.624: Output 'adv_file_output': stopping 11:50:33.624: Output 'adv_file_output': Total frames output: 911 11:50:33.624: Output 'adv_file_output': Total drawn frames: 898 (1522 attempted) 11:50:33.624: Output 'adv_file_output': Number of lagged frames due to rendering lag/stalls: 624 (41.0%) 11:50:33.624: Video stopped, number of skipped frames due to encoding lag: 36/918 (3.9%) 11:50:33.636: ==== Recording Stop ================================================ 11:50:55.116: encoder 'recording_h264': Cannot set the scaled resolution while the encoder is active ```

Steps to Reproduce

1. Launch OBS
2. Start recording (with display capture on to see when it'll freeze, but it can work with game capture as well for example)
3. Change monitor refresh rate in windows settings, you may have to do this a few times until the preview in OBS goes black and doesn't recover.
4. Notice that the UI doesn't freeze immediately, for example the audio mixer is still moving, but if you click on "Stop Recording" then it freezes.

Additional information

The recording is properly saved and all, so there seem to be some error handling that works, but ends up with OBS in an undefined state perhaps.

[WizardCM]

11:50:33.617: [jim-nvenc] nvenc_encode_tex: nvEncEncodePicture failed: 4 (NV_ENC_ERR_INVALID_DEVICE)

The encoder crashed, so at this point OBS should have stopped the recording and displayed an error on screen automatically. Are you saying that didn't happen? You're right that the recording stopped correctly, at least.

If the preview went black, it's possible that the renderer doesn't handle the change properly and causes other things to lock up.

[nstepien]

at this point OBS should have stopped the recording and displayed an error on screen automatically

No error displayed from OBS, no.

[WizardCM]

Alright, I did two tests.

The first was with the default settings with CBR 2500 kbps, and while the encoder really wasn't happy, it didn't end up crashing. https://obsproject.com/logs/lANI4UwTaOqzkg-5

Second test had the settings closer to yours, and I even added a Display Capture source to try and reproduce your conditions. No issues there either https://obsproject.com/logs/OiJnIY8e4lIEGxS2

I did notice that occasionally a window on my main display would teleport to another screen momentarily while the refresh rate changed, and other times Microsoft Edge would hang for a couple of seconds, but both seemed to recover.

The only thing I can think of is that you only have one display, and when there's no display the encoder might get confused. I'll have to test that another time.

I do have one question - which refresh rates were you toggling between? I toggled quickly between 60 and 90 Hz (as 95Hz is as high as my main display can safely go).

[nstepien]

I was toggling between 144Hz and 60Hz. I can try other refresh rates if you want.

It's interesting that the encoder is tightly coupled with the capture part, you'd think it'd record a black screen if nothing can be captured, a bug with nvenc perhaps? 🤔

[WizardCM]

It's interesting that the encoder is tightly coupled with the capture part, you'd think it'd record a black screen if nothing can be captured, a bug with nvenc perhaps? 🤔

Actually, they're not related. Reminder, here's what your log file stated:

1:50:33.617: [jim-nvenc] nvenc_encode_tex: nvEncEncodePicture failed: 4 (NV_ENC_ERR_INVALID_DEVICE)

Changing refresh rates causes the display to disappear for a moment. The event of the disconnection might trigger something in the GPU itself to turn off all capabilities as they are no longer required, and cause the encoder to shut down because of it.

Therefore, my guess as to why I couldn't trigger the error is because I have 3 displays, so as the computer still had at least one other display to render to, it never reached the code path that made the GPU entry "invalid" for the encoder.

[nstepien]

I found a clue!

I'm playing Sekiro and OBS has the same issue (FromSoft please), and I once got a BSOD when trying to terminate the OBS task. The BSOD had the VIDEO_SCHEDULER_INTERNAL_ERROR id, so I turned off Hardware-accelerated GPU scheduling, rebooted, and OBS hasn't frozen since then.

https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/bug-check-0x119---video-scheduler-internal-error

[nstepien]

I ran into this issue again in Elden Ring and OBS 27.2.1. Turning off hardware-accelerated GPU scheduling fixes it once again. Though it is surprising as I believe Elden Ring uses a framerate limited instead of changing the monitor's refresh rate.

[RytoEX]

I cannot reproduce this on Windows 11, with Hardware-accelerated GPU Scheduling (always enabled on Windows 11), with OBS Studio 28.0.3, using Game Capture with Dark Souls III, on a 144 Hz display in a laptop system. Display Capture also seems fine, though I did not test that as thoroughly.

If you are still experiencing this issue, please comment with a new OBS log of a session where this occurred from OBS Studio 28.0.3 (or newer).

[nstepien]

Tried both Sekiro and Elden Ring and I could not reproduce the issue anymore 👍 I guess an Nvidia driver update fixed it at some point.

[RytoEX]

Thanks for confirming!