GPU offloading and shader precaching

[sabian2008]

Hi

I've looked everywhere on internet and searched this bug trucker with tons of keywords without finding an answer to this simple question.

How does shader precaching work when paired up with offloading, in particular NVidia offloading? I can think of three scenarios

1. I start steam on the iGPU but then I add __NV_PRIME_RENDER_OFFLOAD=1 %command% (and, eventually, the other GLX/VK flags) to the launch options of a certain game, do shaders get precompiled for my NVidia card?
2. I start steam on the iGPU but then I add gamemoderun %command%, with the following environment variable defined GAMEMODERUNEXEC=env __NV_PRIME_RENDER_OFFLOAD=1 env __GLX_VENDOR_LIBRARY_NAME=nvidia env __VK_LAYER_NV_optimus=NVIDIA_only to the launch options of a certain game, do shaders get precompiled for my NVidia card? (`gamemoderun is used as an example here, any other script that indirectly sets the appropriate envvars should do it).
3. I start steam with __NV_PRIME_RENDER_OFFLOAD=1 steam and play everything using the dGPU, do shaders get precompiled for the NVidia card?

Sorry for the inconvenience, but I've been wondering this for months, and the only hits I found on Google were people with the same question who couldn't get a definitive answer. This is probably not an issue but a question addressed to @kisak-valve or anyone else with the technical knowledge of how is precaching implemented. I hope that by asking here, the question can serve other people.

P.S: If it serves for something, I am running Fedora 33 with the latest stable drivers from rpmfusion and using 1650 SUPER with an i7 4790K.

[kisak-valve]

Hello @sabian2008, I do not have any insight beyond what you can observe on your own. That said, I would not expect per-game launch options to influence the fossilize child process when replaying .foz to warm the video driver's on-disk shader cache. My assumption would be that case 1 and 2 would not influence shader pre-caching, but case 3 could.

While fossilize is running to replay shaders, you should be able to use htop, find the running fossilize process, then press e to view the environment variables of the process, and look for the PRIME offload related variables. That should be a decent indicator which video driver should be in use.

[sabian2008]

Thanks a lot for the answer. I tested 3. yesterday and I could see fossilize processes running on the dGPU, but that is expected, as they are children of a process running on dGPU (steam). Next time I get a shader precaching I will look at the envvars to see if I notice something useful.

Will write back if I find anything worth reporting.

Is precaching support for offloading setups planned? Trying to answer the question myself, I guess offloading should be officially supported via GUI options (as I've seen discussed in another issue) first.

[kakra]

As far as I know from discussion with Hans in the fossilize bug tracker, the GPU isn't utilized anyways by the shader pre-compile, so nothing gets offloaded to it. It's purely done in CPU. But the cache is dependent on the driver implementation, thus the fossilize process would need to know which driver it has to use. I think this question may be better placed in the fossilize bug tracker. I may have seen a note somewhere that it processes shaders for all GPUs available but I'm not sure.

The discussion about CPU and GPU utilization was part of the system overloading thread where I contributed some ideas and patches to improve the behavior, and the issue has been fixed since then - thanks to a great implementation by Hans.

This may be related: https://github.com/ValveSoftware/Fossilize/issues/110

[philipl]

My anecdotal experience is that even with steam running on the dGPU, shader conversion is targeting the iGPU, based on how it did a full conversion run after I switched from only using the dGPU to a hybrid off-load configuration.

[Root-Core]

@kisak-valve One interesting question that came to my mind is, which version of the shader chache is downloaded? The iGPU is recognized as my system's GPU by Steam. So I would expect Steam to download the Shaders accordingly?

There should be more options in the GUI.. maybe max. threads / RAM and the index of GPU in question.

[stephematician]

Is there any official word on this issue? I'm suspicious that shader pre-caching is not working in most games that are run via offloading - I get a lot of 'stutters' when I enter new areas in games but they disappear if I reload and play through again.

[Root-Core]

I can't say anything definitive, but in my files are nvidiav1 folders.

~/.steam/steam/steamapps/shadercache/[...]/nvidiav1/GLCache/[...]/[...]/steam_shader_cache.bin

The fossilize process has no env variable containing "prime", but it wouldn't be relevant anyways, as the process does not utilize the GPU.

There are env variables for AMD and Nvidia, but only files for Nvidia. I'm mildly positive that it works as intended.

Furthermore, the shader_log.txt has NVIDIA all over the place:

Processing NVIDIA cache file
Writing NVIDIA cache of 1 buckets
Done with merging potential NVIDIA fragments.
Finding NVIDIA cache files, base directory:

This is with an Intel/Nvidia combination. Steam recognizes the Intel GPU in the system information.

EDIT: I've looked into the fossilize code and it seems to pick the first GPU reported by the OS, if no index is specified. So it might differ on different machines / kernel or driver versions.

EDIT 2: Steam actually sets --device-index 0 - I'm not sure if it always sets 0, or if it actually selects the dGPU.