[Linux] Unusually long startup time

ghost commented 8 years ago

This isn't a major issue by any means, but it would be really nice if it could get looked into. Ever since moving over to linux full time, I've had an issue with CS:GO taking an unusually long time to launch.

I've just timed it, and when I launch the game from Steam, it takes a full 10 seconds before the system enters fullscreen mode, and then there is a solid black screen (I have -nosplash -novid set) for another 13 seconds. I don't recall having any sort of delay like this on Windows.. a few seconds at most.

So from launching the game to actually getting to the main menu, there is a total of a 23 second wait time. Here is a screen recording: https://www.youtube.com/watch?v=L5yuYx0zxp4

I'm running the game on a 2.5GB/s Samsung 950 Pro NVMe SSD.. it should be firing up pretty fast.

CS:GO is the only game I own that has this issue.

I would appreciate if someone could take a look. Thanks!

Kernel: 3.19.0-30-generic x86_64 (64 bit) Desktop: KDE Plasma 4.14.13 Distro: Linux Mint 17.3 Rosa Mobo: Gigabyte model: Z170X-Gaming 3 Bios: American Megatrends v: F5h date: 12/23/2015 Quad core Intel Core i7-6700K (-HT-MCP-) cache: 8192 KB Card: NVIDIA GK110 [GeForce GTX 780] Display Server: X.Org 1.15.1 drivers: nvidia (unloaded: fbdev,vesa,nouveau) Resolution: 2560x1440@60.0hz GLX Renderer: GeForce GTX 780/PCIe/SSE2 GLX Version: 4.5.0 NVIDIA 361.18 Card-1 NVIDIA GK110 HDMI Audio driver: snd_hda_intel Sound: ALSA v: k3.19.0-30-generic Card-2 Intel Sunrise Point-H HD Audio driver: snd_hda_intel Card: Qualcomm Atheros Killer E220x Gigabit Ethernet Controller driver: alx IF: eth0 state: up speed: 1000 Mbps duplex: full HDD Total Size: NA (-) ID-1: /dev/sda model: WDC_WD1001FALS size: 1000.2GB ID-1: / size: 406G used: 124G (33%) fs: ext4 dev: /dev/nvme0n1p5

caphene commented 3 years ago

+1

Game takes a long time to start. Strace shows a bunch of timeouts waiting for a mutex.

futex(0x1eed50c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1612663944, tv_nsec=708573000}, FUTEX_BITSET_MATCH_ANY) = -1 ETIMEDOUT (Connection timed out)
futex(0x242da68, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x1eed50c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1612663944, tv_nsec=710700000}, FUTEX_BITSET_MATCH_ANY) = -1 ETIMEDOUT (Connection timed out)
futex(0x242da68, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x1eed50c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1612663944, tv_nsec=712842000}, FUTEX_BITSET_MATCH_ANY) = -1 ETIMEDOUT (Connection timed out)
futex(0x242da68, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x1eed50c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1612663944, tv_nsec=714963000}, FUTEX_BITSET_MATCH_ANY) = -1 ETIMEDOUT (Connection timed out)
futex(0x242da68, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x1eed50c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1612663944, tv_nsec=717117000}, FUTEX_BITSET_MATCH_ANY) = -1 ETIMEDOUT (Connection timed out)
futex(0x242da68, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x1eed50c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1612663944, tv_nsec=719255000}, FUTEX_BITSET_MATCH_ANY) = -1 ETIMEDOUT (Connection timed out)
futex(0x242da68, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x1eed50c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1612663944, tv_nsec=721375000}, FUTEX_BITSET_MATCH_ANY) = -1 ETIMEDOUT (Connection timed out)
futex(0x242da68, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x1eed50c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1612663944, tv_nsec=723495000}, FUTEX_BITSET_MATCH_ANY) = -1 ETIMEDOUT (Connection timed out)
futex(0x242da68, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x1eed50c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1612663944, tv_nsec=725620000}, FUTEX_BITSET_MATCH_ANY) = -1 ETIMEDOUT (Connection timed out)
futex(0x242da68, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x1eed50c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1612663944, tv_nsec=727764000}, FUTEX_BITSET_MATCH_ANY) = -1 ETIMEDOUT (Connection timed out)
futex(0x242da68, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x1eed50c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1612663944, tv_nsec=729884000}, FUTEX_BITSET_MATCH_ANY) = -1 ETIMEDOUT (Connection timed out)
futex(0x242da68, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x1eed50c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1612663944, tv_nsec=732001000}, FUTEX_BITSET_MATCH_ANY) = -1 ETIMEDOUT (Connection timed out)
futex(0x242da68, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x1eed50c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1612663944, tv_nsec=734134000}, FUTEX_BITSET_MATCH_ANY) = -1 ETIMEDOUT (Connection timed out)
futex(0x242da68, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x1eed50c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1612663944, tv_nsec=736264000}, FUTEX_BITSET_MATCH_ANY) = -1 ETIMEDOUT (Connection timed out)
futex(0x242da68, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x1eed50c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1612663944, tv_nsec=738397000}, FUTEX_BITSET_MATCH_ANY) = -1 ETIMEDOUT (Connection timed out)
futex(0x242da68, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x1eed50c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1612663944, tv_nsec=740516000}, FUTEX_BITSET_MATCH_ANY) = -1 ETIMEDOUT (Connection timed out)
futex(0x242da68, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x1eed50c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, {tv_sec=1612663944, tv_nsec=742656000}, FUTEX_BITSET_MATCH_ANY) = -1 ETIMEDOUT (Connection timed out)
futex(0x242da68, FUTEX_WAKE_PRIVATE, 1) = 0

This could be a red herring but these messages stop right before the menu comes up.

Additionally I'm seeing Connecting to CSGO network for a minute or so once I've reached the menu.

turrini commented 3 years ago

I've got the same problem. Startup takes 3-4 minutes now (before was 30 seconds) on NVMe.

I've traced it and it is something related to NVIDIA's shader cache generation. It's a file with 16MB that keeps being recreated every time the game is launched, and they take ages to rebuild:

Folder: $HOME/.steam/debian-installation/steamapps/shadercache/730/nvidiav1/GLCache/11f4fad42593a5e170c906e3fdc05dd1/1dd495eeca3ab130

Files:

steamapp_merged_shader_cache.bin (16mb)
steamapp_merged_shader_cache.toc (496k)

I'm using the same Debian Stable, without driver updates. This problem started to occur after one of the first updates this February.

turrini commented 3 years ago

Well, I disabled the steam shader caching and the problem has gone away, I've followed these steps: https://www.technipages.com/how-to-disable-shader-pre-caching-in-steam

caphene commented 3 years ago

Well, I disabled the steam shader caching and the problem has gone away, I've followed these steps: https://www.technipages.com/how-to-disable-shader-pre-caching-in-steam

This ended up working for me as well.

Shader pre-cache disabled: 25s Shader pre-cache enabled: 3m 19s

+rep @turrini

jthistle commented 3 years ago

It's a shame, because I would really like it if shader pre-caching worked on Linux, but it seems so broken (and has been, for a long time).

samunders-core commented 3 years ago

For me disabling shader pre-cache was not enough, I also had to explicitly remove/rename $HOME/.steam/debian-installation/steamapps/shadercache/730 directory to get sub-minute CS GO startup. i5 3500K, NVIDIA GTX 670, driver 450.66, Debian Buster, kernel 5.4.106-1-pve

mannnish commented 3 years ago

Well, I disabled the steam shader caching and the problem has gone away, I've followed these steps: https://www.technipages.com/how-to-disable-shader-pre-caching-in-steam

I guess we can close the issue, as this worked perfectly fine in most of the cases.

eiglow commented 3 years ago

I have had this bug for years. Shader pre-caching disabled, .steam/steamapps/shadercache/730 is nonexistent. I have an AMD RX Vega 56, on Fedora 34. CS:GO takes several minutes to start up, and by the time it finally starts, it will be taking >5GB of RAM.

Mailaender commented 3 years ago

I can confirm this on @openSUSE with @nvidia 390.144 drivers. Disabling the shader pre-cache improves it a lot.

yuyuen72 commented 2 years ago

confirm this on Ubuntu 20.04 with Nvidia 390.144 drivers, only disabling the shader pre-cache nothing changed, with remove $HOME/.steam/debian-installation/steamapps/shadercache/730 improves it a lot.

kisak-valve commented 2 years ago

Hello, https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18484 looks like something worth testing for Intel (GPU) and AMD (GPU) players. Does not affect NVIDIA players.

ernstp commented 2 years ago

Interestingly there was a update that should improve startup performance released today also: https://blog.counter-strike.net/index.php/2022/09/39497/

pawelpotoniec commented 2 years ago

time reduced from 45 seconds to 2-3

Janhouse commented 2 years ago

Hello, https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/18484 looks like something worth testing for Intel (GPU) and AMD (GPU) players. Does not affect NVIDIA players.

Tried it, it does work. Now just waiting for it to end up in a stable release. Mesa v22.2 doesn't have it and you have to build from git.

tilderain commented 1 year ago

For me disabling shader pre-cache was not enough, I also had to explicitly remove/rename $HOME/.steam/debian-installation/steamapps/shadercache/730 directory to get sub-minute CS GO startup. i5 3500K, NVIDIA GTX 670, driver 450.66, Debian Buster, kernel 5.4.106-1-pve

Make sure you remove $HOME/.nv/GLCache/ as well if it exists. Mine was using the files in there instead of the ones from steam's shadercache

Guanran928 commented 1 year ago

Can confirm this on multiple devices: macOS Big Sur (iMac 2017) ~3min macOS Big Sur (iMac 2019) ~2min Ubuntu 22.04.2 LTS (Lenovo LEGION Y7000 2019, nVidia 1050ti) ~6.5min, also can't enter any map. (stuck progress bar, waited for 30min)

All of them are running on my external TOSHIBA HDD, formatted as ExFAT, with NO issues on Windows 10 Pro.

(should be a software issue since CS:S works perfectly)

SS:

ghost commented 1 year ago

This is still happening to me on Manjaro w/ Nvidia (started about a week or two ago)... wouldn't be a big problem if the game didn't also crash when I open the damn Steam overlay.

EDIT: Also happens with cache deleted from both steam and nvidia directories and the shader pre-cache disabled...

eiglow commented 1 year ago

For me, -vulkan helped a lot with both the initial startup and the loading duration when joining a server.

However, since the dxvk version in csgo is quite old, it was pretty stuttery until I updated it with the latest dxvk-native version from here: https://github.com/doitsujin/dxvk/releases

ghost commented 1 year ago

@eiglow Thanks for the tip! Last time I tried, Vulkan was stuttery for me too. Does updating with the latest dxvk completely remove the stuttering? Do I have to do it every time csgo receives an update? Can I get VAC banned?

eiglow commented 1 year ago

Does updating with the latest dxvk completely remove the stuttering?

It completely removed the stuttering for me. The only downside is that I was unable to change my resolution. This isn't a problem for me on my desktop, but on my laptop with a very high-DPI display and integrated graphics, its a bit more of an issue.

Do I have to do it every time csgo receives an update?

I don't think so unless the update involves that specific file. I've had my game updated a few times since doing it and it seems to still work.

Can I get VAC banned?

No, VAC bans only target cheats. If you're not trying to install a cheat, you won't get VAC banned. The worst that will happen if your files aren't in a consistent state is you can get errors like "VAC was unable to verify your game session", which is fixed either by restarting the game or verifying the files. But I haven't encountered this as a result of upgrading dxvk.

ipaqmaster commented 1 year ago

This seems to come and go over the years and this is another round.

The best/fastest/easiest thing to test would be older gpu drivers (and kernel versions if needed to build). If I find time over the next few weeks I'll test for a solution.

ghost commented 1 year ago

I forgot to say, Vulkan worked for me for like half a match and then become a 5fps slideshow, although 75fps was shown by cl_showfps 1 at all times. So I turned back to OpenGL... but surprisingly the game launched as fast as it did with Vulkan and kinda stayed that way since...

@ipaqmaster I haven't reverted to an older kernel or GPU driver so I doubt the version is what matters. Maybe after the update some cache or files that were meant for the old versions were not properly cleared or updated, and that triggers this pathological case.

ipaqmaster commented 1 year ago

I forgot to say, Vulkan worked for me for like half a match and then become a 5fps slideshow

This is unfortunately a known problem (https://github.com/ValveSoftware/csgo-osx-linux/issues/2901) making -vulkan non-viable for many at the moment though in that issue people are having good luck using LD_PRELOAD="" %command% as their launch options to avoid those stutters. May be worth you trying that as well.

@ipaqmaster I haven't reverted to an older kernel or GPU driver so I doubt the version is what matters

My intention without context was to try downgrading to previous month versions to see if the issue goes away.

Maybe after the update some cache or files that were meant for the old versions were not properly cleared or updated, and that triggers this pathological case.

With that theory I could certainly try pacstrapping to a new zfs root, boot into that and try installing some boring base minimal base packages plus Steam like a brand new Arch user just to confirm whether or not the problem goes away in an environment without historical files laying around. I'll add this to the list.

ipaqmaster commented 1 year ago

[Findings for my case and TL;DR at bottom]

Testing a new rootfs

I created a new rootfs dataset with ZFS and bootstrapped packages to it with pacstrap, those packages included NetworkManager, lightdm, xfce, gnome, steam, nvidia-dkms, zfs-dkms and other userspace utilities as needed or desired. Set a password and booted into it, made myself a user, logged into steam and downloaded CS;GO.

Steam processed shaders first (!) and after 5 minutes of waiting for that, the game launched in this fresh and lightweight Archlinux installation immediately. Like under a second after the window appeared.

I must comment - Steam spent 3-5 minutes processing shaders before opening the game. This is probably the issue many are experiencing as my computer's Steam client hasn't been offering to pre-process shaders at all lately despite the option being enabled in my host's Steam Beta client.

This confirms that a brand new install with the latest drivers, kernel and packages have no problem opening running CS;GO. Which is what my real rootfs install is running too. Next I'll poke at my desktop's filesystems to see if I can magically cure this problem by deleting something, or otherwise.

Continued troubleshooting

Next up in this troubleshooting session I noticed CS;GO would launch faster on my PC after reaching the menu at least once despite this slow startup problem persisting whenever I go to play with my friends every evening or so.

Block level diffing to find a lead as quickly as possible

Rolling back my /data dataset to the previous day featured the 60s CS;GO startup delay again which gave me an interesting troubleshooting idea. I rolled it back again and took a zfs-snapshot for every 10 seconds that CS;GO was stalling to reach the main menu then diffed them. Comparing the zfs-diff's it turns out my CS;GO is building its own shadercache this entire time... and with only a single cpu thread that's where the startup delay is coming from.

[Click to reveal growing shadercache file between diffs while the game launches]

``` me@pc ~ $ ls -alh /data/.zfs/snapshot/*/steam/steamapps/shadercache/730/nvidiav1/GLCache/96c16aaad04bf94d0fca459bd20dd5b6/587d66573b417fc8/* | sed 's/jared/me/g' # -rw-r--r-- 1 me me 242K Jul 27 11:45 /data/.zfs/snapshot/csgotesting3_game_loading_0s/steam/steamapps/shadercache /730/nvidiav1/GLCache/96c16aaad04bf94d0fca459bd20dd5b6/587d66573b417fc8/steamapp_shader_cache0.bin -rw-r--r-- 1 me me 3.1K Jul 27 11:45 /data/.zfs/snapshot/csgotesting3_game_loading_0s/steam/steamapps/shadercache/730/nvidiav1/GLCache/96c16aaad04bf94d0fca459bd20dd5b6/587d66573b417fc8/steamapp_shader_cache0.toc -rw-r--r-- 1 me me 6.0M Jul 27 11:45 /data/.zfs/snapshot/csgotesting3_game_loading_10s/steam/steamapps/shadercache/730/nvidiav1/GLCache/96c16aaad04bf94d0fca459bd20dd5b6/587d66573b417fc8/steamapp_shader_cache0.bin -rw-r--r-- 1 me me 14K Jul 27 11:45 /data/.zfs/snapshot/csgotesting3_game_loading_10s/steam/steamapps/shadercache/730/nvidiav1/GLCache/96c16aaad04bf94d0fca459bd20dd5b6/587d66573b417fc8/steamapp_shader_cache0.toc -rw-r--r-- 1 me me 12M Jul 27 11:45 /data/.zfs/snapshot/csgotesting3_game_loading_20s/steam/steamapps/shadercache/730/nvidiav1/GLCache/96c16aaad04bf94d0fca459bd20dd5b6/587d66573b417fc8/steamapp_shader_cache0.bin -rw-r--r-- 1 me me 23K Jul 27 11:45 /data/.zfs/snapshot/csgotesting3_game_loading_20s/steam/steamapps/shadercache/730/nvidiav1/GLCache/96c16aaad04bf94d0fca459bd20dd5b6/587d66573b417fc8/steamapp_shader_cache0.toc -rw-r--r-- 1 me me 17M Jul 27 11:45 /data/.zfs/snapshot/csgotesting3_game_loading_30s/steam/steamapps/shadercache/730/nvidiav1/GLCache/96c16aaad04bf94d0fca459bd20dd5b6/587d66573b417fc8/steamapp_shader_cache0.bin -rw-r--r-- 1 me me 34K Jul 27 11:45 /data/.zfs/snapshot/csgotesting3_game_loading_30s/steam/steamapps/shadercache/730/nvidiav1/GLCache/96c16aaad04bf94d0fca459bd20dd5b6/587d66573b417fc8/steamapp_shader_cache0.toc -rw-r--r-- 1 me me 25M Jul 27 11:46 /data/.zfs/snapshot/csgotesting3_game_loading_40s/steam/steamapps/shadercache/730/nvidiav1/GLCache/96c16aaad04bf94d0fca459bd20dd5b6/587d66573b417fc8/steamapp_shader_cache0.bin -rw-r--r-- 1 me me 75K Jul 27 11:46 /data/.zfs/snapshot/csgotesting3_game_loading_40s/steam/steamapps/shadercache/730/nvidiav1/GLCache/96c16aaad04bf94d0fca459bd20dd5b6/587d66573b417fc8/steamapp_shader_cache0.toc -rw-r--r-- 1 me me 32M Jul 27 11:46 /data/.zfs/snapshot/csgotesting3_game_loading_50s/steam/steamapps/shadercache/730/nvidiav1/GLCache/96c16aaad04bf94d0fca459bd20dd5b6/587d66573b417fc8/steamapp_shader_cache0.bin -rw-r--r-- 1 me me 103K Jul 27 11:46 /data/.zfs/snapshot/csgotesting3_game_loading_50s/steam/steamapps/shadercache/730/nvidiav1/GLCache/96c16aaad04bf94d0fca459bd20dd5b6/587d66573b417fc8/steamapp_shader_cache0.toc -rw-r--r-- 1 me me 39M Jul 27 11:46 /data/.zfs/snapshot/csgotesting3_game_loading_60s/steam/steamapps/shadercache/730/nvidiav1/GLCache/96c16aaad04bf94d0fca459bd20dd5b6/587d66573b417fc8/steamapp_shader_cache0.bin -rw-r--r-- 1 me me 151K Jul 27 11:46 /data/.zfs/snapshot/csgotesting3_game_loading_60s/steam/steamapps/shadercache/730/nvidiav1/GLCache/96c16aaad04bf94d0fca459bd20dd5b6/587d66573b417fc8/steamapp_shader_cache0.toc -rw-r--r-- 1 me me 48M Jul 27 11:46 /data/.zfs/snapshot/csgotesting3_game_loading_70s/steam/steamapps/shadercache/730/nvidiav1/GLCache/96c16aaad04bf94d0fca459bd20dd5b6/587d66573b417fc8/steamapp_shader_cache0.bin -rw-r--r-- 1 me me 194K Jul 27 11:46 /data/.zfs/snapshot/csgotesting3_game_loading_70s/steam/steamapps/shadercache/730/nvidiav1/GLCache/96c16aaad04bf94d0fca459bd20dd5b6/587d66573b417fc8/steamapp_shader_cache0.toc -rw-r--r-- 1 me me 52M Jul 27 11:46 /data/.zfs/snapshot/csgotesting3_game_loading_80s/steam/steamapps/shadercache/730/nvidiav1/GLCache/96c16aaad04bf94d0fca459bd20dd5b6/587d66573b417fc8/steamapp_shader_cache0.bin -rw-r--r-- 1 me me 229K Jul 27 11:46 /data/.zfs/snapshot/csgotesting3_game_loading_80s/steam/steamapps/shadercache/730/nvidiav1/GLCache/96c16aaad04bf94d0fca459bd20dd5b6/587d66573b417fc8/steamapp_shader_cache0.toc # -rw-r--r-- 1 me me 52M Jul 27 11:46 /data/.zfs/snapshot/csgotesting3_game_done/steam/steamapps/shadercache/730/nvidiav1/GLCache/96c16aaad04bf94d0fca459bd20dd5b6/587d66573b417fc8/steamapp_shader_cache0.bin -rw-r--r-- 1 me me 260K Jul 27 11:46 /data/.zfs/snapshot/csgotesting3_game_done/steam/steamapps/shadercache/730/nvidiav1/GLCache/96c16aaad04bf94d0fca459bd20dd5b6/587d66573b417fc8/steamapp_shader_cache0.toc ```

Once the game is able to reach the main menu those files merge becoming steamapp_merged_shader_cache.bin and steamapp_merged_shader_cache.toc in that same subdirectory.

This unusually long startup time issue can be reproduced by dropping all shadercaching efforts for appid 730 (CSGO) with # rm -rfv /data/steam/steamapps/shadercache/730/nvidiav1/GLCache and opening the game again. This delete will also delete historical efforts.

In my case, Steam hasn't been processesing shaders for CS;GO all month and I imagine that's the root cause of this problem. To reproduce the issue feel free to click 'skip' on Steam's shadercache generation to truly reproduce the issue (if yours still works).

The game's black screen delay will be a variable time based on a computer's max single core boost clock speed assuming no other load I suppose. You can watch -d -n1 ls -lahR /data/steam/steamapps/shadercache/730/nvidiav1/GLCache as the game hangs before the load-screen to see csgo's on-the-fly and incomplete cache build up.

Why isn't my Steam offering to process the vulkan shaders before starting the game?

The new rootfs installation test processed them and after that 5 minute processing wait; the game opened up immediately (presumably as a result). My installation doesn't seem to offer this when I launch CS;GO and I remember how badly fossilize_replay would crunch the entire cpu socket with its multithreaded shader-caching when it came out. Today I haven't noticed it because it isn't running...

To try and fix this I went via the top left Steam menu otions into Steam > Settings > Downloads and unchecked and rechecked Enable Shader Pre-caching (4217 MB pre-cached) and launched CS;GO. This successfully caused steam to spend 5 minutes building a full and complete shadercache for CS;GO on my 3900x cpu.

Peeking inside steam/steamapps/shadercache/730/nvidiav1/GLCache/ you can see a the many threaded steamapp_shader_cache bin/toc file pairs growing file pair being processed for each host thread, putting it all together once it's finished. The resulting steamapp_merged_shader_cache.bin generated by fossilize_replay was is 1.1GB (!) instead of csgo_linux64's single-threaded 52MB one it creates every time I go to open the game at night.

With the above paragraph the game launched in about 6 seconds for me. Still not quite as snappy as that fresh install rootfs was able to pull off but still, this fixed the problem for me.

This ginormous file size makes me wonder if a lot of my stuttering this year has come from this exact same problem. I deleted CS;GO's GLCache again and opened it, waiting 60 seconds and ran map de_dust2. The 52MB shadercache grew a few bytes as I ran around for a moment.

I have a gut feeling I've just accidentally discovered the source of my stutters when peaking enemies in MM during the months this bug resurfaces for me every so often (Months/Years in-between) while thinking it couldn't possibly be shader related given the vulkan shader processing steam is supposed to be doing...

I deleted the CS;GO GLCache directory again but Steam did not offer to regenerate it with fossilize_replay when I launched CS;GO again, causing the 60s black-screen launch delay.

Next I tried exiting Steam and deleting the appid's entire cache path with: #rm -rfv /data/steam/steamapps/shadercache/730/ and opening Steam again then launching the game... Steam still isn't offering to regenerate the shader cache.

Finally I tried deleting the homedir cache with #rm -rfv ~/.cache/nvidia/GLCache/ and that didn't trigger the rebuild either.

GLCache and its UUID subdir system?

All those GLCache subdirectories seem to be hypenless uuid's and don't seem to tie to anything on the system. I assume they're randomly generated and Steam is starting the game without checking if it needs to generate a new shadercache .bin for CSGO before each run.

This is probably where it's up to Valve to permanently fix. Given my csgo appid has a few uuids in its GLCache shadercache directory all dated different months I suspect the UUID is rolling over every so often and Steam doesn't check if it needs to generate a new shadercache.bin for CS;GO.

The UUID may be influenced by new package versions, steam versions, driver versions, randomly or otherwise - causing CS;GO to look in a new/empty nvidiav1/GLCache uuid subdirectory, find nothing and generate its own incomplete one every time I go to play it this month leading to that 60s start delay plus stutters when I peak into an enemy and something loads in (followed by the .bin growing...)

This theory tracks the real issue here: Steam not offering to build a shadercache as-needed nor on-demand when I deleted the CS;GO's shadercahce directory.

The problem is reproducible if you delete appid 730's shadercache directory. The real issue is that my Steam is not checking if it needs to build a shadercache with the Processing Vulkan shaders (xx%) window so if you do get that popup, skip it to verify the issue or leave it to temporarily solve the issue for yourself until the next GLCache UUID shift.

Steam for one reason or another is not rebuilding the cache when there is none leaving it up to the game, slowly, at startup. The game's self-generated shader-cache is only a 52MB.bin compared to Steam's 1.1GB.bin and is probably where my stutters came from around the same time the game started taking a long time to open.

I'm glad to have a lead on the cause of this at least in my case. The game is generating a new 52MB shadercache for itself because Steam isn't generating the full one for the game. every evening I open it to play with friends. This file is significantly smaller than the 1GB shadercache Steam should be generating (And with multiple threads). So I expect this same issue is also the cause of the stuttering I've experienced lately right as this slow launch problem popped up.

Findings and Summary

In my case Steam isn't generating the complete shadercache for CS;GO and the game is launching, seeing no shaders and generating its own incomplete shadercache file which is the reason the game takes 60 seconds for my PC to open and pins a cpu core to 100% for that duration. I suspect given the nature shaders, this is also why my initial peak's and fights with an enemy on a map causes the game to microstutter/pause briefly while it handles more shaders.

TL;DR & Potential workarounds / What worked for me

Be wary of shadercache stutters if you're affected by this slow menu bug. The shadercache it generates is only ~50MB rather than the 1GB shadercache Steam is supposed to be generating for it.
Delete steam/steamapps/shadercache/730/nvidiav1/GLCache/ then open Steam. Top left of the client go to Steam > Settings > Downloads and uncheck/recheck Enable Shader Pre-caching then launch the game. It will re-download the work it needs to complete (~700MB for CS;GO) and when you launch the game it should start generating them. My 3900x took 5 minutes to generate a complete 1.1GB shadercache bin then CS;GO launched in about 6 seconds.
I wonder if dxvk2.0 async can help the game access its main menu while it bakes instead of exclusively waiting for baking to finish? (See: #3123)

ipaqmaster commented 1 year ago

Its been a few days now and CS;GO/Steam have been doing well with a consistent 6-10 second launch time after a few additional seconds of Steam doing vulkan shader preparation against the 1GB worth it already prepared. The game consistently opens in ~6-10 seconds when the game is launched from my steam library and as an added bonus, the game doesn't stutter at all anymore given the 1GB shader cache Steam generates vs the 50MB one the game usually does on its own for 60 seconds every launch.

Something that's probably noteworth is that I deleted my steam-made csgo.desktop shortcut just in case it were causing Steam to skip its multi-threaded vulkan shader generation solution. I've been launching the game directly from the Steam library, though I don't explicitly recall shaders being generated by Steam when launched this way in the past. TBD.

Overall, dropping my Steam's 4GB of Steam's shadercaching efforts by going to Steam > Settings > Downloads and unchecking/rechecking Enable Shader Pre-caching and also removing CS;GO's caches by running rm -rfv /data/steam/steamapps/shadercache/730 before launching CS;GO that one time has made Steam prepare 1GB of vulkan shaders and then consistently prepare them for a few seconds before every launch. In short, I have more or less resolved the problem in my case, though I'm curious why the new rootfs was starting the game in <1s but my PC still has a 6s delay. (Still a large improvement compared to 1 minute per launch...)

Again though, I also no longer using the steam-generated csgo.desktop shortcut to open the game - It is theoretically possible that the Steam Browser Protocol that steam game shortcuts invoke with could've been causing the any% Steam multithread shader skip as well. I haven't tested for that case so it has to be kept in mind until eliminated.

If anyone feels like trying the above shader-dropping trickery this please let me know if it fixes the problem for you. Even if transiently. Further testing/evidence could provide a huge bump for Valve officially solving why Steam silently skips shader generation before the game launches.

ipaqmaster commented 1 year ago

Unfortunately -vulkan still stutters during gameplay and started building its own steamapp_shader_cache0.bin implying this discussion will not solve #2901 nor help with #3172 where -vulkan solves the crashing problem (but stutters... not a good tradeoff...)

At least not yet.

orithecapper commented 1 year ago

-vulkan seems to be better for me :(

ipaqmaster commented 1 year ago

Its been a while since my last comment but since the above actions in my larger comment the game has been consistently launching in ~6 seconds. Some nights I catch steam preparing with all 24 of my cpu threads.

I don't know why Steam stopped preparing shaders before gameplay a few times this decade but that whole reset shader-cache download procedure did the trick. At least until the bug happens again.

It should've clicked for me earlier but is still just a guess - csgo_linux64 was probably synchronously generating the shaders it needed to render everything involved for the main menu - causing the pause. Avoided by letting Steam generate a full 1.1GB shadercache file (Plus any in-game stuttering).

I wonder if something trivial like temporarily running out of disk space causes Steam to stop generating them right before game launch. Will check that later in case its relevant.

Unfortunately I still can't use -vulkan which gives unplayable gameplay stutters. Somehow. I'll have to try the dxvk2.0 async launch parameter some time soon to test whether it solves this problem and also -vulkan's stutter problem on my hardware.

E: It's 1w6d since this comment and as I opened CS;GO tonight; Steam spent about 4 minutes making new shaders for the game. Then it sat on 100% for 30 seconds while it merged them. The game took over a minute to launch.

I ran rm -rfv /data/steam/steamapps/shadercache/730/nvidiav1/GLCache before pkill steam and relaunching didn't trigger a rebuild. Game took over a minute again.

I then went to Steam > Settings > Downloads again to uncheck/recheck Enable Shader Pre-caching (4236 MB pre-cached) and waited for Steam to download the ones for CS;GO again. Once those were done I launched the game and Steam made new shaders. But this time, it took another minute but then subsequent launches were in 5 seconds. I watched the files under steamapps/shadercache/730 recursively and saw Steam merge them after the game exited.

Not sure why this is still an ongoing problem and I'm disappointed this happened again. But the workaround of regenerating seems to be consistent.

ValveSoftware / csgo-osx-linux