Dota 2 forever stuck in bootscreen after loosing vulkan device when VK_EXT_graphics_pipeline_library is used

[mkjOoB]

Your system information

• System information from steam (Steam -> Help -> System Information) in a gist:https://gist.github.com/mkjOoB/4c95c65f5f81a27682853a4c892e4d75#file-systeminfo-log
• Have you checked for system updates?: Yes
• Are you using the latest stable video driver available for your system? No, vulkan-beta
• Have you verified the game files?: Yes

Please describe your issue in as much detail as possible:

Describe what you expected should happen and what did happen. Please link any large pastes as a Github Gist.

Since the "Battle Report" dota 2 update on june 9 2022, vulkan's VK_EXT_graphics_pipeline_library extension support was added. Now game fails to load with GTX 1080 and -vulkan, when having vulkan-beta drivers supporting this extension installed. VK_EXT_graphics_pipeline_library is not yet in stable driver branch. Tested on Linux with nvidia-drivers 515.49.15 and windows 10 LTSC 1809 with nvidia-drivers 517.22

On Linux Steam was launched with: PRESSURE_VESSEL_WRAP_GUI=1 GAME_DEBUGGER=gdb STEAM_LINUX_RUNTIME_LOG=1 STEAM_RUNTIME=1 steam -console -silent -single_core

The game was ran with steam-linux-runtime debbuging gui enabled, tested both with Steam Runtime 2 (soldier) 0.20220803 and Steam Runtime 3 (sniper) 0.20220119.0 which gave no visible difference. That's as per: https://github.com/ValveSoftware/steam-runtime/blob/master/doc/reporting-steamlinuxruntime-bugs.md#advanced-debugging

Logs were collected by selecting "Run interacting shell" set to "Instead of running the command" and then typing "$@" | tee /tmp/dota2.log

When running the game with -vulkan without VK_EXT_graphics_pipeline_library, it works.

My take is: As far as I've notised, the issue persist on Windows and Linux, proton or native, with STEAM_RUNTIME=1 or by running ./dota2 directly bypassing SteamLaunch (which is how I ussually run the game with -gl being on wayland). Either VK_EXT_graphics_pipeline_library is not ready in the game or in the drivers, or is not supported on my gpu generation -- Nvidia's Pascal. I wasn't able to find anything regarding VK_EXT_graphics_pipeline_library on NVidia's public dev forum. It would be interesting to know whether behaviour differse on Turing or Ampere GPUs or with MESA whom are still figuring VK_EXT_graphics_pipeline_library out. GTX 1080 is my only card someone should retest.

Steps for reproducing this issue:

1. Run dota 2 on Linux with -vulkan when running nvidia drivers (only vulkan-beta branch for now) with VK_EXT_graphics_pipeline_library extension support.
2. The game will open a window with a working cursor over dota 2 logo splashscreen and will load forever.
3. Observe vulkan device lost errors in the terminal, see log.

[danginsburg]

Thanks for the detailed bug report. You are getting a VK_ERROR_DEVICE_LOST which indicates a GPU hang. A question for clarification:

As far as I've notised, the issue persist on Windows and Linux, proton or native, with STEAM_RUNTIME=1 or by running ./dota2

You are saying that you see the same issue on Windows. Which NVIDIA driver is that with? You are using 515.49.15 on Linux, the latest Windows Vulkan beta driver is 517.22.

[danginsburg]

Oh, nevermind, I see you said:

Tested on Linux with nvidia-drivers 515.49.15 and windows 10 LTSC 1809 with nvidia-drivers 517.22

Hmm...OK, we'll have to see if we can repro.

[danginsburg]

Not repro'ing for me here on Win10 517.22 GTX 2080. Can you try adding -vulkan_disable_graphics_pipeline_library to your launch options just to confirm it is indeed VK_EXT_graphics_pipeline_library support that is causing the issue?

[mkjOoB]

I retested with -vulkan_disable_graphics_pipeline_library and without. Adding -vulkan_disable_graphics_pipeline_library makes the game work. Retest with -vulkan -vulkan_disable_graphics_pipeline_library: https://gist.github.com/mkjOoB/1f549bb1fc9b98244e1150573efc0527#file-dota2-with-vk_ext_graphics_pipeline_library-disabled-linux-log Retest with just -vulkan, again same error: https://gist.github.com/mkjOoB/1f549bb1fc9b98244e1150573efc0527#file-dota2-with-vk_ext_graphics_pipeline_library-enabled-linux-log

I also fired up my windows 10 LTSC 1809 and it seems that -vulkan_disable_graphics_pipeline_library makes vulkan work.

I would like to get similar logs for windows but idk how to log dota 2 on windows, there's also no minidump generated in Steam/dumps/ since the game does not crash and must be killed.

I wonder if this repro's on other gtx 1080 card or rather all pascal 10 series cards.

[mkjOoB]

Ok, so I managed to get dota2's logs for windows. Here's one for '"C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\cdb.exe" -g -G -o %command% -vulkan' Log: https://gist.github.com/mkjOoB/1f549bb1fc9b98244e1150573efc0527#file-dota2-with-vk_ext_graphics_pipeline_library-windows10-log and '"C:\Program Files (x86)\Windows Kits\10\Debuggers\x64\cdb.exe" -g -G -o %command% -vulkan -vulkan_disable_graphics_pipeline_library' Log: https://gist.github.com/mkjOoB/1f549bb1fc9b98244e1150573efc0527#file-dota2-without-vk_ext_graphics_pipeline_library-windows10-log

[mkjOoB]

Please let me know if there's more I can do to help.

[pdaniell-nv]

@mkjOoB would you mind trying the latest beta driver from here: https://developer.nvidia.com/vulkan-driver. Many thanks.

[mkjOoB]

Here's retest with updated nvidia drivers 517.57 on windows 10, same story, though now with validation layers enabled. Used the standard preset in vkconfig from vulkan 1.3.224 sdk with synchronization stuff enabled, too. Log: https://gist.github.com/mkjOoB/1f549bb1fc9b98244e1150573efc0527#file-dota2-retest-with-synchronization-validation-vk_ext_graphics_pipeline_library-windows10-log

And a Linux retest with latest nvidia drivers 515.49.19 and standard vulkan validation layer settings without additions, same dead hang. Log: https://gist.github.com/mkjOoB/1f549bb1fc9b98244e1150573efc0527#file-dota2-retest-with-standard-validation-vk_ext_graphics_pipeline_library-linux-log

[pdaniell-nv]

@mkjOoB many thanks for the follow up and the logs. I think we may have found the issue and will follow up again when the solution is available. In the meantime you'll need to use the non-beta NVIDIA drivers to play Dota2. Sorry for the inconvenience.

[multiprecision]

Can confirm with today's GeForce driver release 522.25. Program's not responding during start-up. VK_EXT_graphics_pipeline_library is already in the stable channel.

[dimon222]

I'm 1080 ti user and I'm seeing the same issue with newer driver stable 526.47. This flag solves it for me -vulkan_disable_graphics_pipeline_library

[starwavex]

Adding -vulkan_disable_graphics_pipeline_library in game launch options works for me. (I got gtx1080 gpu with 520.56.06 nvidia linux driver)

[Squall-Leonhart]

This extension is now available in the release drivers as of 526.47, and the issue is present there.

[dimon222]

To exacerbate problem, the issue is present even on Windows and Dota is getting rid of OpenGL based on latest announcement. I wonder where will all stuck users go. And no, not this random GitHub thread.

[jaccarmac]

I experienced the same issue on a Quadro 2100M, 470.x driver series. The launch flag works. Because it's an old card, I have drivers >= 471 masked.

[tmzhuang]

On linux 6.0.6, nvidia driver version 520.56.06-8. Found this issue because my game never loaded after recent updates, but passing in -novid -console -high -prewarm -map dota to launch properties worked for me.

[KGB333]

To exacerbate problem, the issue is present even on Windows and Dota is getting rid of OpenGL based on latest announcement. I wonder where will all stuck users go. And no, not this random GitHub thread.

yeah. I'm on Windows and since the driver 522.25 update I can no longer load the game without Adding -vulkan_disable_graphics_pipeline_library in game launch options. BTW the issue is only present on older hardware like gtx series cards (mine is TITAN Xp). And I have no issue to launch the game on a RTX platform (2070s).

[Squall-Leonhart]

Fullscreen Exclusive also breaks the game with similar crashes on these new drivers, if you are on 526, the new Layered on DXGI Swapchain option gets you back into game, the Native setting is causing crashes with FSE.

False alarm, just a corrupt vulkan pipeline cache.

[bricss]

Same stuff here:

• Platform: Dell XPS 15 9560
• OS: Win11 22H2
• Graphics: GeForce GTX 1050
• Driver: GeForce Game Ready Driver 526.61 hf

Game doesn't start w/o -vulkan_disable_graphics_pipeline_library flag and hang-up on load screen

[Strykar]

Noticed the issue since the last driver update on Arch Linux (nvidia 520.56.06) on a GTX 950 and 1080. Adding -vulkan_disable_graphics_pipeline_library fixes the issue and Dota goes past the splash screen and runs fine.

Sysinfo - https://gist.github.com/Strykar/7e5a05c1c252b38adc6d574ba3dfa9fb

[danginsburg]

I believe we have a fix for this coming in the next Dota update. The root cause of the issue is described here: https://github.com/KhronosGroup/Vulkan-ValidationLayers/issues/4768

[danginsburg]

This update has shipped. Please let us know if it has fixed the issue for you.

[Squall-Leonhart]

the framerate is trash with this update.

[jaccarmac]

I can confirm that the issue is fixed with an update. I'm not experiencing the framerate drops others are, either.

[KGB333]

I can confirm that the issue is fixed with an update. I'm not experiencing the framerate drops others are, either.

same here, everthing is good

[starwavex]

After the update, the game loads up to visible game background, but then hangs. Sometimes it shows the main menu and instantly hangs. Yet I'm able to move the cursor, although the interface doesn't respond. Otherwise I just see the background image. Tried completely cleaning vulkan cache - no success. Still only if I add -vulkan_disable_graphics_pipeline_library the issue vanishes. GTX 1080 + nvidia (linux) 520.56.06

[mkjOoB]

@danginsburg Hmm, still broken for me somehow. That's still on 520.56.06 GTX 1080. Tested on wayland and xorg sessions. Removed all the flags but -vulkan. Goes past the dota 2 logo but then displays gradient background for the main menu and stays like this while spewing out same culprit device lost error which seems to be forever. I even used a distribution's precompiled kernel for retesting this, shouldn't be my system's configuration problem. The replay is from sway's wayland session. I purged game files and reinstalled before major testing. Also steam vulkan shader caching settings doesn't seem to change a thing, I usually go without it being on nvidia, I've turned it on and had it do the full rebuild after reinstall just for testing sake. I'll test bit later, if it's still doesn't work under windows 10.

https://user-images.githubusercontent.com/82988916/200928391-a34e0571-f61a-42e2-8bce-915747717bc9.mp4

dota2-still-broken-linux.0.log

[mkjOoB]

@KGB333 @jaccarmac @starwavex Are you sure that's with -vulkan and not with -vulkan -vulkan_disable_graphics_pipeline_library? This should only repro on nvidia pre turing cards.

[KGB333]

@KGB333 @jaccarmac @starwavex Are you sure that's with -vulkan and not with -vulkan -vulkan_disable_graphics_pipeline_library? This should only repro on nvidia pre turing cards.

100% sure, run the game nice and smooth with my Titan Xp.

[starwavex]

@KGB333 @jaccarmac @starwavex Are you sure that's with -vulkan and not with -vulkan -vulkan_disable_graphics_pipeline_library? This should only repro on nvidia pre turing cards.

@mkjOoB I'm sure. And I experience exactly the same what you've recorded on video.

[bricss]

Works for me, except preload phase got a bit slower, but fps seems to be good 🎃

[Strykar]

Confirming my GTX 1080 running on driver 520.56.06, even post Dota patch, will not work without vulkan_disable_graphics_pipeline_library. This time it goes past the splash screen and the game hangs on the main menu screen.

[Squall-Leonhart]

Confirming my GTX 1080 running on driver 520.56.06, even post Dota patch, will not work without vulkan_disable_graphics_pipeline_library. This time it goes past the splash screen and the game hangs on the main menu screen.

delete your pipeline caches.

[Strykar]

delete your pipeline caches.

How?

[jaccarmac]

This should only repro on nvidia pre turing cards.

Not sure if you mean the original issue or the fix here, but.

Are you sure that's with -vulkan and not with -vulkan -vulkan_disable_graphics_pipeline_library?

All my tests before this comment were without an explicit -vulkan in the launch options. I can't really test the pre-patch behavior with the flag, but now the game launches fine with and without it.

[mkjOoB]

@danginsburg I just updated to newly released 525.53 nvidia-drivers. Steam has automaticly processed shaders before game launch and the issue is still present. Everything as in my last video: https://github.com/ValveSoftware/Dota-2-Vulkan/issues/413#issuecomment-1309340999

Though here's another log with these validation variables set with new drivers VK_LAYER_ENABLES=VK_VALIDATION_FEATURE_ENABLE_SYNCHRONIZATION_VALIDATION_EXT:VALIDATION_CHECK_ENABLE_SYNCHRONIZATION_VALIDATION_QUEUE_SUBMIT:VK_VALIDATION_FEATURE_ENABLE_BEST_PRACTICES_EXT VK_INSTANCE_LAYERS=VK_LAYER_KHRONOS_validation

dota-still-broken-after-fix.1.log

[rodlorric]

Hi, I'm experiencing the similar issues.

Your system information

• System information from steam (Steam -> Help -> System Information): System Info

• Have you checked for system updates?: Yes

• Are you using the latest stable video driver available for your system?: Yes

• Have you verified the game files?: Yes

Please describe your issue in as much detail as possible: Since the last update of the nvidia drivers (526.86) the game hangs after a few seconds while running with the vulkan renderer (-vulkan). Adding the -vulkan_disable_graphics_pipeline_library option, the game runs without any issue. Additionally the game ran fine with the previous version of the driver.

Describe what you expected should happen and what did happen: The game should not hang with Vulkan renderer.

Steps for reproducing this issue: Run the game with -vulkan option. Wait a few seconds.

Additional info: WinDbg Log with -vulkan, without -vulkan_disable_graphics_pipeline_library: ERROR! VK call failed! result = VK_ERROR_DEVICE_LOST on lines 1080 through 1358. WinDbg Log with -vulkan, with -vulkan_disable_graphics_pipeline_library

Edit: Mixed up the gists links.

[mkjOoB]

With today's dota 2 update, I no longer have this problem :+1: No more hangs, VK_EXT_graphics_pipeline_library is in use! Here's a log to back it up: dota2-now-fixed-with-graphics-pipeline-library.log

Thanks for fixing this @danginsburg, please keep this https://github.com/ValveSoftware/Dota-2-Vulkan/issues/282#issuecomment-1304064179 in mind, I'm not the only one who'd be really happy to see it fixed, seems like the time with -gl getting purged with next major update. Have a nice day.

This issue is probably fixed for you too @starwavex, @rodlorric, if you mind testing again so we can close this.

[danginsburg]

Great, the latest update contained another fix for not using too many UBOs in the pipeline layout, which sounds like it was the source of your hang.

[starwavex]

@mkjOoB I can confirm that the latest update (build id: 9946186) solved the issue for GTX1080, game runs successfully (at least tested for few hours) both with and without precached Vulkan shaders. Much thanks @danginsburg for the feedback!

[rodlorric]

I can confirm that after the last update the game is no longer crashing and VK_EXT_graphics_pipeline_library is in use! 👍 Thanks @mkjOoB for you bug report and @danginsburg for fixing it!

[Strykar]

Confirmed working on a 1080 now without vulkan_disable_graphics_pipeline_library, thanks!

[danginsburg]

OK, I'm going to close out this issue, thanks for all the responses (and for your patience, there was a series of very tricky issues here).

[bricss]

Surprisingly, now game hangs up & freeze w/o -vulkan_disable_graphics_pipeline_library right at hero selection screen in diretide mode.

• Platform: Dell XPS 15 9560
• OS: Win11 22H2
• Graphics: GeForce GTX 1050
• Driver: GeForce Game Ready Driver 526.98