[LINUX] Incredibly slow map load time on fast system.

[oskarstr]

Your system information

System information

• Have you checked for system updates?: [Yes/No]

In short: Manjaro (latest), Threadripper 3970x, AMD 6800XT

Please describe your issue in as much detail as possible:

Loading maps, even in offline with bots is taking 3-5 minutes on average. It's stuck either at Initializing World or Loading resources.

In task manager there's 1 single cpu thread during the hang at 100% until the match starts.

Game itself runs perfectly fine at 300+FPS.

I wonder if I can access some logging to find out what exactly it's loading or getting stuck at during that time but I'm not sure how to enable some advanced debugging.

What have I tried:

• Tried it on Ubuntu 20.4 with same system - same problem exists.
• I've tried the latest stable official drivers, also trying the mesa-git drivers with no change
• Tried almost every launch option that referenced similar issues online, nothing helps
• Changing graphics settings to low, disabling MSAA, multicore rendering etc. No dice.
• Tried disabling shader cache in steam. No dice. In fact even the steam shader cache just sits at 0kb so not sure if it works to begin with.

Some Observations:

• Tried it on MacOS with same system - problem doesn't exist. Could that be driver problem?
• Got dealt same map few times in a row while playing wingman. Randomly, the map will load right away with no 3-5 min time but there's no pattern - it wasn't that first time map loads slow and second time it was fast. It was random and maybe twice.

Steps for reproducing this issue:

1. Launch Game
2. Start an Offline match or join live server
3. Get stuck for 3-5 during Loading Resources or Initializing World

[kisak-valve]

Hello @kokozaurz, this reads like the game is waiting on mesa/radeonsi to compile shaders and the video driver's on-disk shader cache isn't helping speed up that process on subsequent runs for some reason.

[oskarstr]

Thanks for the reply @kisak-valve!

If i understand correctly, in ideal situation this behaviour normally should be just experienced once, and past that the shaders would be loading from cache - correct?

If that's the case, could it be that the shader cache isn't created in the first place? or perhaps there are issues reading the created shaders?

I have tried with steam shader cache on and off with no change as mentioned before, the steam shader cache just stays at 0kb regardless.

Is there anything you can suggest me to try to debug this?

[kisak-valve]

Steam's shader pre-cache support is supposed to help by warming up the video driver's on-disk shader cache before games are run in the first place. That is to avoid encountering stutter and longer load times on the first run of a game.

Enabling/disabling the shader pre-cache support setting in Steam should have no effect on the video driver's usage of their own on-disk cache, just the cache warming process done by Steam.

[oskarstr]

@kisak-valve thanks.

So if I'm reading it correctly, most likely the issues are related to the mesa driver vs csgo?

• Is there a way to disable something on csgo side to confirm that?
• Is there a way to get some in detail debugging log from csgo during the time frame when it hangs?
• can you suggest anything else to try?

thank you.

[kisak-valve]

It might be interesting to move/remove <Steam library folder>/steamapps/shadercache/730/mesa_shader_cache and/or [...]/mesa_shader_cache_sf which would purge the game-specific on-disk shader cache for mesa, but I'd expect that to get invalidated every time you changed mesa driver versions or distros.

[oskarstr]

@kisak-valve tried suggestions. Here are my findings:

• <Steam library folder>/steamapps/shadercache/730/mesa_shader_cache 730/mesa_shader_cache part doesn't exist
• deleted mesa_shader_cache_sf, however after attempting to launch the game, the directory doesn't get recreated. there's also a directory called mesa_shader_cache (without the _sf) that gets accessed during load that I can see with inotifywait -m command and here's an example output from it:

./.cache/mesa_shader_cache/5c/ OPEN 6d3f880b825609e5a851ce4ddfe035ef53d64e ./.cache/mesa_shader_cache/5c/ ACCESS 6d3f880b825609e5a851ce4ddfe035ef53d64e ./.cache/mesa_shader_cache/5c/ CLOSE_NOWRITE,CLOSE 6d3f880b825609e5a851ce4ddfe035ef53d64e ./.cache/mesa_shader_cache/08/ OPEN 1ef82e0eb8387ec389bf3b25685eca3a42d37e ./.cache/mesa_shader_cache/08/ ACCESS 1ef82e0eb8387ec389bf3b25685eca3a42d37e ./.cache/mesa_shader_cache/08/ CLOSE_NOWRITE,CLOSE 1ef82e0eb8387ec389bf3b25685eca3a42d37e ./.cache/mesa_shader_cache/ec/ OPEN 15a4d2d5baae6052390506d527b536517f49d6 ./.cache/mesa_shader_cache/ec/ ACCESS 15a4d2d5baae6052390506d527b536517f49d6 ./.cache/mesa_shader_cache/ec/ CLOSE_NOWRITE,CLOSE 15a4d2d5baae6052390506d527b536517f49d6

[oskarstr]

another interesting observation: overwatch session loads up right away. However, it's bugged out, the person is stuck in the same looping position in 'warmup', even though i hear the game happening in background and scoreboard is getting updated.

[oskarstr]

After ALOT of different trial and error I can confirm that I found the solution to the problem.

The solution: I moved the GPU from PCIe slot 1 -> slot 2

All of a sudden I'm loading into maps right away and also fps has become more stable and increased by about 10-15%.

Some extra information for those coming from google looking for solution:

I did have this as a Proxmox VM with the GPU passed through, and to eliminate the VM environment as a possible culprit I created a linux drive and booted from that to have native bare metal install. Problem was, in VM I had no issues launching game, but on bare metal I couldn't even launch the game in Manjaro until I installed the Flatpak version of steam. Now I managed to launch the game but still… 3-5minute load time. Before I gave up, I thought might as well swap out the pcie slots (if the other guy fixed it by changing monitor, worth a try) and luckily it did fix it.

And before you ask whether bios had slow speed set up for the pcie slot 1 - no, they're all full speed and no different than one another. For reference, the motherboard is ASUS ROG STRIX TRX40-E Gaming.

[blogdron]

You resolved problem but it not resolve real problem. You're just in luck. =) Changing the slot entailed deleting the cache or creating a new one.

For the rest, first check if the problem is fixed when disabling the cache, to do this, set this line in the game launch parameters

MESA_GLSL_CACHE_DISABLE=true %command%

It it not work set this

if you have more more more more more100500 AAAAAAAA+N+1 games try 20G 30G 50G

MESA_GLSL_CACHE_DISABLE=false MESA_GLSL_CACHE_MAX_SIZE=10G %command%

This will increase the allowed cache size. If it helped, then the limit was just filled. Just remove all cache and delete launch parameters

[oskarstr]

Thanks @fedor-elizarov

a question on this, wouldn't the cache be cleared anyways since I setup a completely fresh install of manjaro and the problem still did persist?

The only way this would make sense is if you're increasing/or clearing the cache on the GPU itself which would lead to a much bigger problem of it not clearing it.

[blogdron]

@kokozaurz

• 1 wouldn't the cache be cleared anyways since I setup a completely fresh install of manjaro

If you install new system you system is clean, GLSL cache the cache will be created from scratch. But if you reinstall you distributive and saved the partition /home/ or /home/$USER from the previous installation all your user data is saved as well as problems. native cache located this /home/$USER/.cache/mesa_shader_cache/ steam cache locate this /home/$USER/.steam/steam/steamapps/shadercache you can just remove this folders they will be recreated But then there will be a new synchronization, some games may start with a delay at first, since they will need to create a new cache, but then everything will be fine.

• 2 The only way this would make sense is if you're increasing/or clearing the cache on the GPU itself which would lead to a much bigger problem of it not clearing it.

No. The shader cache does not load the memory of the video card and does not affect it in any way, and the cache is not stored in the memory of the video card. Only individual files are taken from the cache on demand. These are just pre-prepared programs for the video card. When you run a game for a video card, many small programs are created, they are called shaders. They can be compiled or in advance then they will just be loaded or they can be compiled for your video card directly by your driver at the time of launching the game. In order not to compile every time you restart the game (after all, your video card does not change every time), then these small programs are saved to files and when you start the game again, instead of compiling the shaders for your video card again, they will simply be ready from cache. These are just prepared files. If you delete them, new ones will simply be created. No need to worry

The problem may not necessarily be cache size. For example, if you somehow changed the permissions on directories so that they cannot be read, then there will also be problems.

I just assumed that the reason is in the limit for the size of the cache.

[oskarstr]

@fedor-elizarov,

Thanks for taking the time to give background on this, it all makes sense.

1. It was completely fresh install and I did start the game couple of times to remove the possibility of this being just the first shader generation. Prior to the fresh install I also did remove mesa_shader_cache and that got recreated. The steam shader cache didn't exist.

2. That's good to know, didn't know that there's nothing stored in GPU memory. I also did think of permissions on shadercache folders and just to be sure, I did try to enable rw permissions to all and nothing changed. Also thought maybe it's an io issue between storage or ram and the system but everything seems correct on that front.

It's still a mystery as to why switching to the pcie slot 2 made this to work but I've submitted an update to mesa devs about this. Funny enough there was another person on mesa git issue tracker that solved identical problem by changing from 60hz monitor to 144hz. The switching of pcie slot as a solution for mesa devs is just as big of mystery as it was for swapping a monitor.

If (hopefully not) the problem comes back, I'll try your suggested commands. For now, not going to touch it if it works :)

[blogdron]

@kokozaurz

you can try this for check in terminal

mkdir mycache

launch parameters csgo

MESA_GLSL_CACHE_DIR=/home/$USER/mycache MESA_GLSL_CACHE_DISABLE=false MESA_GLSL_CACHE_MAX_SIZE=10G %command%

run csgo and

ls -R mycache

If you don't see any files in the output. It means that something is seriously broken.

This is to temporarily override where and what size the cache will be and what size it will be. By the way, if you have a fast drive, then you can override the cache directory by specifying a different location. (But this will not work for games that download the cache themselves)

It's still a mystery as to why switching to the pcie slot 2 made this to work but I've submitted an update to mesa devs about this. Funny enough there was another person on mesa git issue tracker that solved identical problem by changing from 60hz monitor to 144hz. The switching of pcie slot as a solution for mesa devs is just as big of mystery as it was for swapping a monitor.

Yes it is strange

I heard that someone had a bug (or crooked hands with a bad memory of what they were doing in the terminal), well, by default, the cache in the system was disabled, which led to a similar problem of yours. Also, changing the video card slot probably makes MESA think that there is a new device and that means the old cache can no longer be used and you need to create a new one for the new video card. Perhaps this behavior solved the problem, but this is just a guess.

For now, not going to touch it if it works :)

Yes if works do not touch :)

[subanz]

I'm having a very similar issue and I found a potential fix although not a great one.

For me the game takes a long time to start up, map loading takes forever getting stuck on Loading Resources and/or Initializing World.

Loading the same map again also gets stuck on Loading Resources and/or Initializing World, on Windows loading the same map again is very fast.

What works for me is using Steam Native instead of Steam Runtime. It seems some outdated library in the Steam Runtime is causing this issue. In Steam Native the game starts up quickly and loads maps normally also loads previously loaded maps very quickly.

Unfortunately it doesn't seem Valve cares about issues like this, in Team Fortress 2 people are still forced to load a system library to prevent having double sensitivity.

I don't know which library it is but assuming we can figure out the culprit we could keep using Steam Runtime and only preload that specific system library.

That being said I don't know if I have the energy or time to check each library to find out which one is causing this slow loading problem right now.

@fedor-elizarov

I've also tested your suggestion and yes the shaders are generated on disk, don't think this is a shader generation issue maybe some sort of file system problem.

[CaptainCoward]

This is still an Issue. Only fix so far seems to use -vulkan.