Shader Cache is Being Downloaded and Compiled After Every Single Reboot

[gardotd426]

Your system information

• Steam client version (build number or date): Sep 16 2021 at 00:09:09
• Distribution (e.g. Ubuntu): Arch Linux
• Opted into Steam client beta?: [Yes/No] Yes
• Have you checked for system updates?: [Yes/No] Yes

Please describe your issue in as much detail as possible:

Every time I launch Steam after restarting my computer, Steam re-downloads and compiles the shader cache for every game. I have "Allow background processing of Vulkan shaders" enabled, and every time I launch Steam after a reboot, I have several downloads for shader caches, and opening the Settings and going to "Shader Pre-Caching" shows that the shaders are being compiled. After they're done, if I exit Steam and re-launch it without rebooting, nothing happens, but I can reliably reproduce it by simply rebooting.

Before this started a couple weeks ago, this was not an issue, and shaders would be downloaded and compiled only periodically or after an Nvidia driver update. Also, the size of the shader cache has ballooned to 15.5 GB, before it was only about 7 or 8 GB, and it consistently stayed around that size for months and months. I haven't installed any new games since then, the only thing that's changed is Doom Eternal added Ray Tracing and DLSS support.

I don't think this is the same as #6299, as it's not limited to Steam Play titles, it's all games. Also, it doesn't happen every time I launch Steam, only every time I reboot.

All of the updates are between 100 KB and 50 MB, except the Resident Evil 2 Remake which was just 2 GB, and Borderlands 3 which was 4.2 GB. Neither were game updates.

EDIT: After the latest round of downloads the shader cache has jumped up to 18 GB. Weirdly enough though, Disk Usage in the Download Manager shows 0 B/s throughout the downloads:

Steps for reproducing this issue:

1. Join the Client Beta
2. Enable "Allow background processing of Vulkan shaders"
3. Start Steam, download and compile shaders, exit Steam, reboot and relaunch Steam.

Hardware/System Info

Ryzen 9 5900X Nvidia RTX 3090 Nvidia Driver Version: 470.63.01 Arch Linux 5.14.3-tkg-cfs Steam System Info: https://gist.github.com/gardotd426/3aa96697991bcc8d796be45814435a83

[gardotd426]

For a while, when it was seemingly "fixed," I'd still get regular updates every day but they would be 1-2 MB, and the "processing" time for each game would take just a few seconds, up to a minute.

Now it's up to 10-15 minutes for most games. Waframe has been processing for 15 minutes. And it's on 56%.

[NTMan]

@gardotd426 you still have few games installed by my standards.

I have 601 games installed and a daily download package happens every day for 150 games.

But what upsets me most of all is not the download volume, it really may not be a large 1Gb, but how much information has to be saved to the hard disk.

For example, the update for "Prefator: Hunting Grounds" weighs 7.2Mb, but 31.3Gb will be written to the disk (write amplification), and this often happens.

[kisak-valve]

Hello @NTMan, the example you've given is a game content update, not a shader pre-cache update. We're able to tell because currently the disk activity graph at the top of Steam's download view works with game content updates right now, but not shader pre-cache updates.

[jarrard]

In the past I've been able to just hover over the 3 vertical lines icon to see if its a cache update or game/content update.

[NTMan]

Hello @kisak-valve I want to raise awareness of the write amplification issue. This is when the volume of downloaded bytes needed be multiplied by 1000 ... 2000 for understand how many bytes were actually written to the disk. And this is true for shader cache updates as well. It turns out, according to my formula, 500MB of downloaded updates actually wrote 500GB ... 1TB of data to the disk.

Below are three screenshots where my formula works exactly. Unfortunately Steam does not have a separate counter of bytes written to disk.

[gardotd426]

This is getting beyond stupid at this point:

THAT'S AN 8GB SHADER CACHE UPDATE. 24 hours after I had a 3-4GB one.

Borderlands 3 shader pre-caching update system is objectively broken at this point.

[jarrard]

This is getting beyond stupid at this point:

THAT'S AN 8GB SHADER CACHE UPDATE. 24 hours after I had a 3-4GB one.

Borderlands 3 shader pre-caching update system is objectively broken at this point.

When you hover over the (!) it says shader cache?

[gardotd426]

When you hover over the (!) it says shader cache?

Yes of course, I wouldn't be posting this otherwise. It's 100% shader cache.

[gardotd426]

Every single day:

[jarrard]

Yeah I typically see Ark download 1+ GB regularly of 'shader cache'. The steam server is either updating them regularly or can't seem to verify the local data correctly.

Does not happen for all games, but certainly a whole bunch of them.,

[gardotd426]

I get shader cache updates for every game in my library at least once a day (sometimes twice), but Borderlands 3 is the only one that's almost always between 3 and 8GB. There's one more that I can't recall right now that's also pretty regularly like 1GB, but that's it.

Honestly if fossilize let us configure how many cores it uses, so I could use 100% of my 5900X, I wouldn't mind nearly as much. But since it won't use more than like 30-40% of the CPU, once the shader cache downloads finish, I have to then wait for what seems like forever for the shaders to actually compile. If I want to play anything in the meantime I have to disable the background compiling of shader cache option and then re-enable it when I'm done.

[gardotd426]

I've had FOUR 4GB-7GB shader cache updates for Borderlands 3 in the last 18 hours

[deathxxx123]

Hi there, i was too seening that shader updates after new UI update of steam client. Most of the time it was just checking the games for integrity, and that was maybe always same games. Sekiro, Shadow of The Tomb Rider. I don't see other games i have to check for integrity. All others are like 5-15GB games. And i don't see steam checking them for integrity. For a instance i have some games with Lutris, witch have some shader pre-chache files. I never see shader updating process or game check for integrity, and i have game that are 30GB, and always i start it in no time with no checks or updates. Maybe Valve should see something from Lutris.

[gardotd426]

@deathxxx123 You're mistaken here. Some Lutris games distribute a DXVK State Cache with their games. They 'don't distribute all the stuff that Valve does with their shader precaching system, they don't distribute any shader caches/Valves fossilize-created .foz caches.

Not to mention the fact that Valve also already DO provide a DXVK State Cache in their shader precaching if the game uses DXVK (which is the only type of cache Lutris ever distributes).

[gardotd426]

Still more people are reporting it on Reddit

[drevoborod]

Same issue for me: Skyrim Special edition downloads cache of 1.2 Gb every day, mostly several times - after every game restart!

[jarrard]

I think you could experience some cache updates after a system update of kernel or mesa etc.. Which is almost daily for me.

Some of my games still like to regularly download shader cache but its normally 100-200MB which is fine. All I can recommend is check your steam shader cache folders for read and write permissions and also manually flush them.

[gardotd426]

I think you could experience some cache updates after a system update of kernel or mesa etc.. Which is almost daily for me.

Cache updates after driver updates are to be expected. That's not what the bug report is over. The bug report is over daily 1, 2 or even 4GB cache updates (sometimes even more than once a day) for some seemingly "problem" games (like Borderlands 3).

I've seen Borderlands 3 get two 3-4GB shader cache updates in one day. With zero system updates.

All I can recommend is check your steam shader cache folders for read and write permissions and also manually flush them.

You can easily check the size of your entire shader cache for all installed games by going to Steam -> Settings -> Shader Pre-Caching, and it will tell you the size. It being non-zero means that there aren't read/write issues, but more than that, manually deleting your shader cache will immediately trigger Steam to rebuild the entire thing (which if you have a lot of games, can easily be 20+ GB, right now my shader cache is 30GB). It will then require you to recompile all those shaders from scratch, which even on a powerful CPU like a 5900X can take hours (especially since Steam won't allow you to tweak the compilation to use your CPU's full power/all cores).

TL;DR I wouldn't recommend it at all.

[truboxl]

A warning on using Shader Cache may incur increased bandwidth usage is not enough to indicate how broken this feature is.

Steam should have metered connection detection. This supposedly broken feature can easily burn through somebody's monthly data cap...

[drevoborod]

I think you could experience some cache updates after a system update of kernel or mesa etc.

No, no updates - they don't arrive so frequently. I use Manjaro, it updates itself onse in a week or two, but shader cache is being updated once every few hours. So these events are not connected at all.

All I can recommend is check your steam shader cache folders for read and write permissions

Of course, all permissions are correct - it's in .steam folder inside my home folder.

You can easily check the size of your entire shader cache for all installed games by going to Steam -> Settings -> Shader Pre-Caching, and it will tell you the size.

1354 MB. Seems correct to me - I have only two games installed, and one of them (Skyrim Special) downloads 1.2 Gb every time.

[BloodyIron]

I'm still having this issue, what's going on then?

[norbusan]

Same here, Skyrim downloads 1.2G every single day I reboot ... that is crazy.

[VladimirMrzv]

On latest steam update. Shaders Cache downloads not for all installed games, and do full download. Some games cannot be moved upper or down

[jarrard]

1.5GB for The Long Dark? hmmmm.

[vzsolt1981]

Same problem, shader cache is redownloaded and rebuilt for every game every time Steam is started (w/o reboot). And that issue has been going on for years now, without apparent fix.

[KurtGokhan]

I had also opened a thread on Reddit about this bug. https://www.reddit.com/r/SteamDeck/comments/tqzxff/steam_downloads_too_many_shader_precaching/

This is a bigger problem in Deck. I don't want to waste my Deck's precious battery life on downloading shady shader updates over Wifi. Some of these updates can be as big as 800 MB (Disco Elysium and Skyrim SE has the biggest updates as I observed). Also there is no way to disable shader precaches in Steam Deck Game Mode.

[jarrard]

I recently cleared my shader cache then re-enabled it to download the stuff again. But each day I boot up, I can see a re-download of shader cache of up to 4.4GB worth!

Is the shader cache updating every time mesa is being updated? I run mesa-git which updates regularly so maybe that triggers it? I have NO IDEA. But it downloads shader cache continously each day I boot up like its starting from fresh.

I think Valve need to loosen up on the requirements for shader cache updating if it is indeed needing a update per micro git update or sub mesa release. I have no idea how bad it is for NVIDIA users since their drivers don't update as often.

[optlink]

I don't think it has anything to do with Mesa updates. I haven't updated Mesa in weeks and I have been suffering over 10GB of shader downloads each and every day.

[vzsolt1981]

@jarrard I have an NVIDIA GPU, and this issue persists since this feature was introduced.

To add insult to injury, it's all done on an NVME drive, so it's practically burning money away.

Based on the hundreds (if not thousands) of reports that prove it's broken, I'm not sure why this feature is enabled by default in the Linux client in the first place.

[jarrard]

I have a game installed but not setup with a prefix yet called CALL OF JUAREZ GUNSLINGER and it downloads 500-600MB of shader cache each day even tho I have never played it yet.

It seems some games are fine, they download a little bit like a few MB which I can deal with, but some games just want huge giant piles of shader cache downloaded regularly if not daily. Is quite weird.

Call of Juarez ---- 582MB (no prefix) Planetside 2 ------ TBA (no prefix) Space Engineers - 414MB (no prefix) Elite Dangerous -- 263-539MB (no prefix)

These are not once off numbers but a daily routine for these games in regards to shader cache downloading!

Based on the hundreds (if not thousands) of reports that prove it's broken, I'm not sure why this feature is enabled by default in the Linux client in the first place.

Because without shader caching, many games stutter badly and are unplayable for some. Like Elden Ring.

[kisak-valve]

"Fixed an issue where updating graphics drivers would re-download media pre-caching files" in the 2022-03-31 Steam client beta update should help with some of the larger downloads.

[BloodyIron]

"Fixed an issue where updating graphics drivers would re-download media pre-caching files" in the 2022-03-31 Steam client beta update should help with some of the larger downloads.

Progress!

[thepwrtank18]

"Fixed an issue where updating graphics drivers would re-download media pre-caching files" in the 2022-03-31 Steam client beta update should help with some of the larger downloads.

Progress!

"That is the sound of progress, my friend."

[BloodyIron]

Switched to Beta branch and so far does not appear to be downloading quite like it was before, but it is re-rendering shader cache each time I log in, or something along those lines. Annoying...

[NTMan]

I currently have a steam version from Mar 31 2022 23:11:12.


And this update did not help me much. I want to use the example of one game to show how often it is updated by Steam.

1. steam updates the game Rise of the Tomb Raider - Apr 2 13:31


2. 9 hours later, steam updates Rise of the Tomb Raider again - Apr 2 22:17


3. after 9 hours steam updates again Rise of the Tomb Raider - Apr 3 07:26


4. 11 hours later, steam updates Rise of the Tomb Raider again - Apr 3 18:01


5. 2 hours later, steam updates Rise of the Tomb Raider again - Apr 3 19:56


In total, the game has been updated five times in two days, is this normal?

P.S. Maybe it's not the shaders that are updated, but Steam has problems with updating games for owners of large libraries.

[jarrard]

Pay attention to the shader cache size its downloading (not applying). Mine have reduced considerably which is fine when its just a few kb or few mb. It was the shader cache re-downloads of 300MB-2GB that were puzzling.

[KurtGokhan]

Feels like not much has changed for me too. It still seems to download a lot of updates for the same games over and over. I am in Stable channel. Here is my version info:

There is another weird bug where a game seems completed even though it is still downloading.

And if you click "Clear All", it will be cleared from the list but still keep downloading as seen in the banner.

[Eisenvaddi]

At the moment, every start of Steam i load 15GB of Shaders. Steam on Linux is the slowest Application on Earth.

[Eisenvaddi]

After 3 years, i hoped, Linux will come. No! With Linux, explicit with Steam, you will only have problems but you will not be able to play. You start Steam, you want to play... NO! You have to download 20G of Shaders! Thats the trouth!

[Eisenvaddi]

Maybe ~20GB of Shaders, at the Moment more then 30GB of Shaders with MSFS2020.

[jarrard]

Maybe ~20GB of Shaders, at the Moment more then 30GB of Shaders with MSFS2020.

When was the last time you purged the shadercache folder and started fresh?

[NTMan]

Pay attention to the shader cache size its downloading (not applying). Mine have reduced considerably which is fine when its just a few kb or few mb. It was the shader cache re-downloads of 300MB-2GB that were puzzling.

1. Why update shaders if there is no difference?
2. Why, when updating shaders, show the game changelog of four months ago?

Even if the update size is small, when there are many games on a large slow HDD, it takes a long time.

[BloodyIron]

Maybe ~20GB of Shaders, at the Moment more then 30GB of Shaders with MSFS2020.

When was the last time you purged the shadercache folder and started fresh?

Why doesn't STEAM have a method for this for those that don't know where it exists?

[XapaJIaMnu]

Some debugging: I found that this occurs on my machine where I have an encrypted home drive gets unlocked at every boot and then created as /dev/mapper/home. On my friend's PC who has the steam library at a drive mounted by /etc/fstab but no disk encryption, this doesn't happen. It does happen if he removes the /etc/fstab entry and manually mounts it in a new directory every time. I suspect what this means is that steam detects if the drive or the mount folder is new and something bad happens. On the other hand with drives/mount points that don't change no redownloads occur.

[optlink]

I am not sure if this is related to the above but I have my Steam library in a ZFS dataset and thus no fstab entry. I get huge downloads every time I start Steam and they get larger the more time passes between starts of Steam.

[KurtGokhan]

My whole library with 32 installed games got a shader update today. I wonder if Valve is not able to reproduce this issue, or if they don't care. This is a highly disturbing issue and should be top priority imo.

[jarrard]

I still get daily shader cache updates but most are only a few MB in size which is fine. I don't think I have too many games doing big continuous shader updates, maybe max size atm is 35MB.

The shader caching system I believe is dynamically and continuously updated on Valves end so some of the shader files will probably experience daily updates to cover more driver/compiler versions..

I really don't know what the shader cache servers look like but I presume their a archive of pre-compiled shaders per driver per kernel/compiler/driver versions. So quite large at Valves end.

[STiAT]

Still, without kernel, driver or proton changes it downloads shaders every other day, often 3-4 times a day. That also requires the shader cache to be rebuilt for those games (at a library of 36 games installed, about 20 update daily, some more than once a day).

That makes me wonder - how do they identify those new downloads? I deem it unlikely that they find and precompile certain shaders more than once a day especially on games where we have not seen game updates in ages (DA:O, The Witcher 1&2, Mount&Blade: Warband etc.).

I've cleared my whole shader cache and let it recompile, still facing this issue. I think they have some issue with the validation, thus re-downloading stuff they don't need to, and within that triggering the re-building.

[KurtGokhan]

That makes me wonder - how do they identify those new downloads? I deem it unlikely that they find and precompile certain shaders more than once a day especially on games where we have not seen game updates in ages (DA:O, The Witcher 1&2, Mount&Blade: Warband etc.).

My theory is that new downloads happen because of changes to any field of the game. That includes store information like prices, tags, description etc. Maybe that causes the hash to change and that is why Steam tries to download entire shader cache again. Sound stupid but as is this bug.

[STiAT]

Replying to https://github.com/ValveSoftware/steam-for-linux/issues/8076#issuecomment-1114264947

Actually, it does not download the entire cache again, it downloads some small chunk (at least since the fix of the steam client), but that still seems to trigger processing the whole shaders.

Anyway, something must happen that the cache gets deemed invalid in some parts at least, at the what this could be .. your guess is probably as good as mine since we don't really know how the validation process works.