Video freezes, sunshine dies on reconnect of stream

[Stono]

Is there an existing issue for this?

• [X] I have searched the existing issues

Is your issue described in the documentation?

• [X] I have read the documentation

Is your issue present in the nightly release?

• [X] This issue is present in the nightly release

Describe the Bug

Hi, Running the current nightly (14-05-2023), and trying to play diablo IV - server slam. Sunshine will randomly crash, usually when going between zones in the game.

The specific steps are:

1. Launch Playnite
2. Start Diablo
3. Pretty quickly within the game the video stops working (but can still hear the sound).
4. Try and disconnect/reconnect the stream, doesn't work
5. Sunshine dies Fatal: Hang detected! Session failed to terminate in 10 seconds.

I see an event in the windows event viewer:

Faulting application name: Sunshine.exe, version: 0.0.0.0, time stamp: 0x645d9e36
Faulting module name: Sunshine.exe, version: 0.0.0.0, time stamp: 0x645d9e36
Exception code: 0x40000015
Fault offset: 0x0000000000051ad6
Faulting process ID: 0x0x870
Faulting application start time: 0x0x1D986780641A859
Faulting application path: C:\Program Files\Sunshine\Sunshine.exe
Faulting module path: C:\Program Files\Sunshine\Sunshine.exe
Report ID: 0138d7a0-f25b-437f-84c9-6fdcfaf114d8
Faulting package full name: 
Faulting package-relative application ID: 

I can make this happen fairly consistently, so if there are specific logs or debug steps i can take to provide more information - please let me know.

It doesn't seem to happen in other games which I find interesting.

One thing that is certainly different about Diablo IV is that currently the game is forced into Windowed Borderless mode. The rest of the games i play are fullscreen.

Expected Behavior

No response

Additional Context

No response

Host Operating System

Windows

Operating System Version

11

Architecture

64 bit

Sunshine commit or version

0.19.1.17124deb225f07c9347778ed867496339dad372f

Package

Windows - installer

GPU Type

Nvidia

GPU Model

4070Ti

GPU Driver/Mesa Version

531.79

Capture Method (Linux Only)

No response

Config

global_prep_cmd = [{"do":"\"C:\\Program Files\\Sunshine\\tools\\start.bat\"","undo":"\"C:\\Program Files\\Sunshine\\tools\\stop.bat\"","elevated":false}]
sw_preset = medium
min_threads = 2
min_log_level = 2
hevc_mode = 0
encoder = nvenc
fps = [60,90,120]
audio_sink = {0.0.0.00000000}.{2d4e313a-84c6-4cbe-b92a-ec3576d31305}
nv_preset = p1
upnp = enabled
resolutions = [
    1920x1080,
    2560x1440,
    3840x2160
]
fec_percentage = 5
qp = 28

Apps

No response

Relevant log output

sunshine.log

[Stono]

Hmmm looks like https://www.reddit.com/r/nvidia/comments/1100nw4/gpu_hardware_scheduling_works_moonlight could be related. i use hardware accelerated gpu scheduling for dlss (its required). if it is that, it's funny really because I play cyberpunk with this configuration just fine.

[Nonary]

HAGS does not cause streams to freeze anymore, you are instead dealing with #1031

Diablo 4 is known for being a VRAM hog on higher settings.

[Stono]

Ahhh well that's gutting. Only just bought the 4070, dismissed the 12gb vram as something that's overhyped, immediately hit a problem because of it haha.

Thanks for confirming my issue isn't HAGS anyway!

[makedir]

Same issue here with a RTX4080, rBAR activated, latest drivers and Win 11, totally not related to VRAM. Always error code 0x40000015 of the crash logged in Windows logs. Stream randomly freezes with audio still working, reconnecting causes sunshine.exe to crash. This randomly happens everywhere mostly OUTSIDE of game and therfor little VRAM usage. For example just opening GFE, or Minecraft launcher and clicking around.

I am 99% sure it is still related to GPU hardware scheduling, and the bug with it is still not fixed (under some conditions). I have set the registry HwSchMode to 1 so far, because the option is not there anymore in Windows 11 to turn it off, not sure if the registry key does anything though.

I have noticed the freeze mostly happens under specific context switch, mostly never in game.

@Stono where do you get you need to enable HAGS for DLSS?

[ReenigneArcher]

@makedir you can still enable/disable HAGS in Windows 11 settings.

[makedir]

@ReenigneArcher No it was hidden for me for whatever reason, it just showed up after I set the registry entry manually. Here someone also said it was hidden for them: https://answers.microsoft.com/en-us/windows/forum/all/where-did-hardware-accellerated-gpu-scheduling-go/28064de6-abbc-4f12-8014-a4a89b3e0ad5

It is a fresh Windows 11 installation. My theory is ,that it is hidden for new Win 11 installations if the registry key is missing, and just shows up if you updated Win 11 through the time where it was still there.

I noticed this already some months ago, that the option was gone / hidden after some Window 11 update.

I will test the next days if the freeze still happens with the registry option to 1 / off.

Sadly you cant use DLSS 3.0 FG with HAGS off... what a pity.

[ns6089]

For example just opening GFE, or Minecraft launcher and clicking around.

Interestingly enough, both of them use chromium.

And when I try to reproduce hags+realtime+vram freezes on desktop, they usually happen when I'm poking around discord (again, chromium).

But hard to say whether it's correlation or causation, chromium is one complex piece of software after all.

[makedir]

@ns6089 The freeze mostly always happened when I was clicking some toggle option or button both in GFE or Minecraft launcher (or maybe changed focus of the windows?). I had no freeze so far playing a game, just outside of games, on Windows desktop in some windows.

[makedir]

I just enabled HAGS again went into GFE and it froze right when I clicked on this button

[makedir]

@ns6089 Could it be that the HAGS freeze bug was fixed in nightly compared to 0.20? I installed nightly yesterday and tested again with HAGS on and had no freeze so far anymore, where the freeze happened actually quickly before with 0.20. Maybe just luck. Will test more.

[ns6089]

To a certain definition of "fixed". Current nightly lowers sunshine gpu scheduling priority, it fixes most freezes but hurts game capture performance rather bad. So that fix is going to be disabled by default and moved to options. I tried a lot of things, and will add some more safeguards, but ultimately the freeze is in driver and we can do nothing about it.

[makedir]

@ns6089 Damn : / how badly does it hurt the performance? More latency, or what is the impact?

[ns6089]

@ns6089 Damn : / how badly does it hurt the performance? More latency, or what is the impact?

It depends on how busy the gpu is. If it's under 100% load, then pretty bad. Larger unstable latency and even decreased capture frame rate.

[makedir]

Could a different capture method help with that (I guess if you could use NVFBC it wouldnt happen?), or use h264 instead of hevc, or lower the NVENC profile type from slow to default for example? does h264 use Nvidia NVENC actually or does it use software encode? i read something about low latency / hevc, but for h264 it said something else in the sunshine settings with ffmpeg I think instead of NVENC. would it be bad to use 264? i think my client renders h264 a little bit faster than hevc, so i switched to h264.

[makedir]

Did anything change with last Nvidia drivers? I get freezing now again with sound working where I had not any issues the last 2 months. And I cant reconnect anymore, Sunshine is dead after the freeze and cant reconnect anymore. Looking on the server no connection shows in Sunshine tray icon, cant reconnect anymore though with Moonlight.

[shayaknyc]

I found this thread because I've been having the same issue. Running nightly v0.21.0.593e170da87fda7d81769273a922f148f73b1768

Here's the output from my windows events viewer:

Faulting application name: sunshine.exe, version: 0.21.0.0, time stamp: 0x659ce1a1
Faulting module name: sunshine.exe, version: 0.21.0.0, time stamp: 0x659ce1a1
Exception code: 0x40000015
Fault offset: 0x0000000000051086
Faulting process id: 0x78ec
Faulting application start time: 0x01da441bd5db6ef2
Faulting application path: D:\Users\[user]\Desktop\Sunshine-Portable-Nightly\sunshine.exe
Faulting module path: D:\Users\[user]\Desktop\Sunshine-Portable-Nightly\sunshine.exe
Report Id: 54c7bbaf-7bea-4cc6-b3c2-737371cf58ef
Faulting package full name: 
Faulting package-relative application ID: 

Running Windows 10 21H2 build 19044.3086

I have a Nvidia RTX 3060ti. Happened to be I was mid Hogwarts Legacy, when the moonlight client just froze on my Nvidia Shield - so I exited, and then reconnected, and that's when sunshine crashed on the desktop I was streaming from.. Any ideas?

Edit 2024-01-11: Disabled HAGS, and now no issues at all. Also updated nightly.

[cgutman]

The crash is expected in this condition. It's because Sunshine detects the deadlock condition and terminates itself to allow it to recover.

If you want to avoid the hangs, turn off HAGS (preferred) or set the option in the Sunshine NVENC tab to not use real-time priority.

This will keep happening until Nvidia fixes their driver. AMD now supports HAGS and they don't hang under these conditions, so it's confirmed to be Nvidia's bug to fix.

[shayaknyc]

If you want to avoid the hangs, turn off HAGS (preferred) or set the option in the Sunshine NVENC tab to not use real-time priority.

Hmm, I already disabled HAGS, but is it better to enable it in Windows, and change the sunshine config to not use real-time priority?

This will keep happening until Nvidia fixes their driver. AMD now supports HAGS and they don't hang under these conditions, so it's confirmed to be Nvidia's bug to fix.

Great.....I'll hold my breath /s

[aacebedo]

For the record, with a 40XX series card and with a game like AW II, there is some big issues. I tried to enable Frame generation to stream in 4k maxed out, the latter requires HAGS to be activated. If I enable the option in sunshine it almost immediately freezes the stream. If I deactivate it, the frame generation seems to be taxing enough to make the stream drop to 15 to 20 frames, it becomes unplayable.

So on heavy games, frame generation seems to be out of the question ATM because of this bug on HAGS