Shader Cache is Being Downloaded and Compiled After Every Single Reboot

[gardotd426]

Your system information

• Steam client version (build number or date): Sep 16 2021 at 00:09:09
• Distribution (e.g. Ubuntu): Arch Linux
• Opted into Steam client beta?: [Yes/No] Yes
• Have you checked for system updates?: [Yes/No] Yes

Please describe your issue in as much detail as possible:

Every time I launch Steam after restarting my computer, Steam re-downloads and compiles the shader cache for every game. I have "Allow background processing of Vulkan shaders" enabled, and every time I launch Steam after a reboot, I have several downloads for shader caches, and opening the Settings and going to "Shader Pre-Caching" shows that the shaders are being compiled. After they're done, if I exit Steam and re-launch it without rebooting, nothing happens, but I can reliably reproduce it by simply rebooting.

Before this started a couple weeks ago, this was not an issue, and shaders would be downloaded and compiled only periodically or after an Nvidia driver update. Also, the size of the shader cache has ballooned to 15.5 GB, before it was only about 7 or 8 GB, and it consistently stayed around that size for months and months. I haven't installed any new games since then, the only thing that's changed is Doom Eternal added Ray Tracing and DLSS support.

I don't think this is the same as #6299, as it's not limited to Steam Play titles, it's all games. Also, it doesn't happen every time I launch Steam, only every time I reboot.

All of the updates are between 100 KB and 50 MB, except the Resident Evil 2 Remake which was just 2 GB, and Borderlands 3 which was 4.2 GB. Neither were game updates.

EDIT: After the latest round of downloads the shader cache has jumped up to 18 GB. Weirdly enough though, Disk Usage in the Download Manager shows 0 B/s throughout the downloads:

Steps for reproducing this issue:

1. Join the Client Beta
2. Enable "Allow background processing of Vulkan shaders"
3. Start Steam, download and compile shaders, exit Steam, reboot and relaunch Steam.

Hardware/System Info

Ryzen 9 5900X Nvidia RTX 3090 Nvidia Driver Version: 470.63.01 Arch Linux 5.14.3-tkg-cfs Steam System Info: https://gist.github.com/gardotd426/3aa96697991bcc8d796be45814435a83

[kisak-valve]

Hello @gardotd426, this reads like the current intended behavior of the Steam client. These shader pre-cache updates were happening before, but with less visibility.

Since you've opened this issue report, we might as well ponder the finer details. Please attach ~/.local/share/Steam/logs/shader_log.txt as a file.

[gardotd426]

shader_log.txt

[jarrard]

Been happening for a month for me. Never ending downloading.

[kisak-valve]

What I'm mostly seeing in the log is:

[timestamp] [AppID <AppID #>] Starting Shader Swarm download job.
[timestamp] [AppID <AppID #>] Target manifest <target manifest #> of size <larger size>.
[timestamp] [AppID <AppID #>] Source manifest <source manifest #> of size <smaller size>u.
[timestamp] Committed bucket <#> (AppID <AppID #>) from <source manifest #> to <target manifest #>.
[timestamp] [AppID <AppID #>] Finished Shader Swarm download job: No Error

These look like legitimate pre-cache delta-patch update, and not a duplicate download.

There's also:

[2021-09-17 03:52:20] Queuing depot manifest 4256548600728534165 (2159391403 bytes) for bucket ID 1027610, type SteamSwarm/G7:CompatVideoTCMediaV1 (AppID 883710).
[...]
[2021-09-17 04:13:54] Committed bucket 1027610 (AppID 883710) from 0 to 4256548600728534165.
[2021-09-17 04:13:54] [AppID 883710] Finished Shader Swarm download job: No Error

[2021-09-17 03:52:20] Queuing depot manifest 1000222499080350120 (4542080587 bytes) for bucket ID 731639, type SteamSwarm/G7:CompatVideoTCMediaV1 (AppID 397540).
[...]
[2021-09-17 04:23:47] Committed bucket 731639 (AppID 397540) from 2385576768023649330 to 1000222499080350120.
[2021-09-17 04:23:47] [AppID 397540] Finished Shader Swarm download job: No Error

To me, Commited bucket # from 0 to <target manifest #> reads like Resident Evil 2 (883710) did full download of transcoded video content, and Borderlands 3 (397540) got an update to the transcoded video it uses with Proton, and that could account for the larger download sizes for those games.

(Speculation) I'm guessing that there's a single file size limit to delta patching with steampipe and the transcoded video .foz files easily exceed that limit, causing a full download in both cases.

Again, important detail here is that these aren't duplicate updates. Steam's pre-caching system is slowly discovering more shaders being used by the games in the wild, and adjustments are being made to the sets of transcoded video to improve in-game video playback of some games.

[gardotd426]

I just got another update for every game in my library, this is the third time today that every game I have has had a shader cache update. Is that really intended behavior?

[jarrard]

Yeah it is 100% broken imo. My cache is almost 10GB and I see it PROCESSING the same games over and over again for months now. This doesn't seem right to me! (My games list has barely changed, I'm not buying games constantly)

[brain-anti-freeze]

Thanks @kisak-valve those log messages are interesting. Grepping my shader_log.txt and shader_log.previous.txt files I can see that only a single game ever downloads deltas (936790: Life is Strange True Colours) all the other ones (36 unique app ids) always download the full shader cache (and none of those contain the 'Source manifest' line). This definitely feels like an issue with the content delivery system not knowing about the existing shader content and mistakenly downloading the whole thing every time.

shader_log.previous.txt shader_log.txt

[jarrard]

I just deleted all my shadercache folder on all my drives and reconsolidate via symlink to the .steam folder. See if that helps, so far it seems ok. I feel steam for linux should just do this automatically...

[folknor]

Same thing here. Since the new download UI was added I've noticed Steam downloads a metric fuckload (that's a technical term) of data every time I start it - so this behavior has probably been present way longer than that. I start Steam once every day after work, and then it runs until I shut down around bedtime.

EDIT: I'll get a screenshot tomorrow or the next day.

The downloads almost never have any information tagged on. I have no idea why Steam does it or what it is. Luckily I don't pay for bandwidth.

https://gist.github.com/folknor/73fcc0b5d229c86af44c4dfe9b409f8a

[folknor]

Here we go, only 18 downloads today, and most of them quite small.

[jarrard]

Just noticed that BFV has a shadercache of 1.9GB!! Some of these games just have massive caches!

[eloj]

(@kisak-valve said): this reads like the current intended behavior of the Steam client. These shader pre-cache updates were happening before, but with less visibility.

Is it intended that the shader downloads are not tagged as such in the downloader? You put in the effort to make it so the download could have a little info icon telling you what sort of update it is, these should clearly be marked as shader updates, but they are not.

"A new (i) icon next to the game's title will reveal a tooltip displaying the types of content that is included in that update. Types consist of: Game Content, Downloadable Content, Workshop Content, and Shader Pre-caching. This icon only appears if the update is not solely game content." -- https://store.steampowered.com/oldnews/88497

Yet the "(i)" icon is nowhere to be seen:

The fact that it's not there, makes it seem like the updates are, by implication, "solely game content" which is clearly false. It would solve a lot of confusion to fix this minor issue.

Once everyone understrand what these updates are, you could go on to perhaps offer an option to filter them out, but for that to work they must be properly tagged.

[Leopard1907]

Also same problem on my end too. RDR2 downloads 141 MB shader cache at every Steam client run. Doesn't happen right away, requires booting the game. After quitting from the title, download begins.

shader_log.previous.txt shader_log.txt

[kisak-valve]

Hello @Leopard1907, looking at your log and described symptoms, I think you're seeing a separate issue from this issue report. Please open a new issue report.

[jazztickets]

The Steam Client Beta from Sept 28 has really helped!

https://steamcommunity.com/groups/SteamClientBeta/announcements/detail/2874976428915672907

[brain-anti-freeze]

I thought it had helped me as I was no longer getting the download after the game exits but after doing a bit more investigation it seems that it still does it but only for the transcoded_video.foz file but not the swarc file. However now most games now no longer downloads the shaders at all (even if I've deleted the directory or I turn shader caching off and back on again) and the amount shown in settings is still reset to 0 every time so the issue changed for me but still not working as expected.

[lucifertdark]

I'm seeing small amounts of files being downloaded but having a custom version of Proton installed stops it in it's tracks.

[brain-anti-freeze]

After digging around a bit more (and turning caching off and on, deleting the cache folders, opting out and back into the beta etc) I now have 12 games which it downloads the caches for on every steam start but they all fail to process with the error Cannot find impl for original cache type 7 from depot. and no other downloads are taking place either in the background or after a game exits. It also now saves and uploads a minidump file on every steam start.

[brain-anti-freeze]

I guess the lack of downloads might be due to the datasets starting over in this new beta and it will take time for the shaders to be recollected and become available for download again. I'll give it a few days and see if I'm getting anything.

[lucifertdark]

My latest log says I've been uploading shaders as well as downloading. The client is still reporting nothing pre-cached but I can live with that as long as the caching is actually happening.

[brain-anti-freeze]

I'm seeing those too but the upload always fails with server busy

[2021-10-01 18:55:16] Register Shader request for 1162ba8b973ec4b6 / L2:b40f80f34cf78f091a6da598b73981d7 shader 0 failed with server reply: Busy [2021-10-01 18:55:16] Register Shader request for 1162ba8b973ec4b6 / L2:b40f80f34cf78f091a6da598b73981d7 shader 1 failed with server reply: Busy [2021-10-01 18:55:17] Register Shader request for 1162ba8b973ec4b6 / L2:b40f80f34cf78f091a6da598b73981d7 shader 2 failed with server reply: Busy [2021-10-01 18:55:17] Register Shader request for 1162ba8b973ec4b6 / L2:b40f80f34cf78f091a6da598b73981d7 shader 3 failed with server reply: Busy [2021-10-01 18:55:18] Register Shader request for 1162ba8b973ec4b6 / L2:b40f80f34cf78f091a6da598b73981d7 shader 4 failed with server reply: Busy [2021-10-01 18:55:18] Register Shader request for 1162ba8b973ec4b6 / L2:b40f80f34cf78f091a6da598b73981d7 shader 5 failed with server reply: Busy [2021-10-01 18:55:18] Register Shader request for 1162ba8b973ec4b6 / L2:b40f80f34cf78f091a6da598b73981d7 shader 6 failed with server reply: Busy

[brain-anti-freeze]

Today's beta update has helped a little though, the Cannot find impl for original cache type 7 from depot. error has been resolved. The 12 caches that it now downloads on every steam client start are at least replaying properly. Unfortunately the initial bug is still present, its downloading the full shader cache for them each time (1.5Gb in total). Some games do now download a cache after they finish (RDR2 does), still the full unchanged file though.

Attached is the result of opening steam, waiting for the downloads and replays to finish then closing and re-opening it (identical downloads and replays happen).

shader_downloads.txt shader_log.previous.txt shader_log.txt

[Plagman]

Can you try today's Client Beta?

[lucifertdark]

I just got the update, it's only downloading shaders for Proton 6.3 & experimental, ignoring all other versions I have installed. This is the message I get for all the others...

Attempting to find current bucket (compat tool: 'proton_316'); temporary dir: /home/user/.local/share/Steam/shader_cache_temp_dir_d3d11_64 [2021-10-02 07:20:40] Unknown shader cache type, ignoring

[brain-anti-freeze]

@Plagman today's beta resolves the downloads issue for me ! I'm only getting the expected downloads and replays now. All uploads still fail with server busy but that could very well be legit since the caching was started over. Thank you everyone who worked on resolving this issue !

[gardotd426]

Myself and others have reported that Borderlands 3 is getting 4-6GB shader precache updates literally every day. That game specifically, and multiple people are experiencing it.

[deathxxx123]

Same thing to me too. After steam client get this new UI update, i got often checking games or download chache. I'm lucky that i play only few games. I imagine if i had my full library, how many time i need to wait every time i stars steam client. With old client all was so much better.

[GithubUser5462]

Every time i restart steam it tries to download a 700MB "Shader pre-caching update" for No Man's Sky.

Verifying the integrity of files moves the game to completed. I am on the latest steam beta.

[gardotd426]

I just watched Borderlands 3 update and process it's shader cache twice in a row. Like literally back to back. And the overall shader cache jumped 3 GB up to 20GB total.

This is going to destroy battery life on the Steam Deck if it stays this way.

[kisak-valve]

Hello @gardotd426, now that the shader pre-cache reset which was connected to the 2021-10-06 Steam client update has had some time to warm up, are you still seeing large shader pre-cache updates with Borderlands 3 on a regular basis?

[gardotd426]

@kisak-valve, not like... two and three times a day, but still pretty regularly. Though in the past couple days I feel like I've seen it less? I'll take the next day or so to actually look and try and measure in a more scientific way and report back.

[folknor]

Here's how it looks for me this morning, after the client being shut down overnight, roughly 10 hours (note that Factory Town actually had a patch, and RimWorld is workshop updates; all the others are weird - they're not workshop or game updates, and they're not marked as shader updates):

This happens every day. The number of games that it happens to changes every day. Obviously I'm on beta, and I realise that most of these updates are tiny in size - I just think they're strange.

[gardotd426]

Oh yeah I'm still getting every game updating at least once a day, for sure. @kisak-valve, I've gotten 2 Borderlands 3 shader cache updates in the past couple hours (one just now), but the download sizes are dramatically smaller (2-3MB vs 1-3GB before). The disk size (like what it says the disk usage will be after downloading) is still like 2GB for each download but anyway, yeah the downloads are way smaller.

Are daily shader cache updates for every game intended behavior? Just not giant multi-gigabyte updates? If that's the case then yeah I think we can close.

[kisak-valve]

Thanks for checking. I was specifically asking about the large, repeated downloads. It sounded like transcoded video for running Borderlands 3 with Proton was changing.

Since Steam's shader pre-cache is crowd sourced, those small updates should be progressively less common shaders seen with the games as they are discovered, independent of game updates.

[gardotd426]

Okay, since the multiple-times-per-day Gigabyte+ downloads seem to have stopped, I guess I'll close this for now. If the issue crops back up I'll reopen or at least comment here. Thanks Kisak.

[NTMan]

We shouldn't have closed the bug report.

On my systems keep downloading updates in size very similar to the shader cache.

shader_log02.txt

shader_log03.txt

shader_log04.txt

shader_log05.txt

shader_log06.txt

shader_log07.txt

What other proof is needed?

[gardotd426]

@NTMan those are all tiny. Most of them are in the kilobyte range, with only a few more than a couple MB, and none that are 100MB or more. Like Kisak said, he was concerned about daily (or multiple times a day) re-downloads of gigabyte-or-more shader cache downloads. It's expected to have regular small updates.

Thanks for checking. I was specifically asking about the large, repeated downloads

As he also says:

Since Steam's shader pre-cache is crowd sourced, those small updates should be progressively less common shaders seen with the games as they are discovered, independent of game updates.

So yeah, the issue can be closed.

[NTMan]

In my case, the total update volume was 1Gb (113 games). Updates that have exceeded 100MB:

• Forza Horizon 5 - 485.5MB

• Mass Effect Legendary Edition - 179.9MB

[gardotd426]

According to your screenshots you have no updates for Mass Effect and only one 11MB update for Forza. The total size from the screenshots is nowhere near 1GB. If you have other updates that weren't in the screenshots, I don't know how anyone is expected to know about those.

Also, more importantly, you have several BRAND new games (some of which don't even work on Linux so I don't know why they're installed, like Battlefield 2042 and Halo Infinite), and as Kisak literally said - the shader cache updates are crowd sourced, and so obviously the newer the game, the more updates you'll get, and the larger they'll be, as more players contribute to the shader cache and it "learns" more. Of course Forza is going to have sizable updates because it's a brand new game.

This is exactly what one would expect for brand new games.

On Sun, Nov 21, 2021 at 11:15 AM Mikhail @.***> wrote:

In my case, the total update volume was 1Gb. Updates that have exceeded 100MB:

• Forza Horizon 5 - 485.5MB
• Mass Effect Legendary Edition - 179.9MB

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/ValveSoftware/steam-for-linux/issues/8076#issuecomment-974847054, or unsubscribe https://github.com/notifications/unsubscribe-auth/AM5Y33YKGRRJP2SZZUPWXTTUNELJXANCNFSM5EGOUWIQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[gardotd426]

@kisak-valve, I believe this actually might need re-opening. I've seen 4 updates to Borderlands 3 in the past 18 hours, and they've all been over 1GB (3 of them were 3-4GB). Once I had 3GB update complete and immediately go right into a 1GB update (I don't mean it completed, then 2 seconds later started a new one. I mean it never even left the "Downloading" section, it immediately completed and started a new shader update).

[Eisenvaddi]

@kisak-valve is an liar.

For months he has said the problem will be fixed. No, even after 3 months it was not fixed.

EVERY time I start Steam, I have to download 20GB of shaders. LINUX! Not Windows. Windows works well.

I am now waiting until February and for the Steam Deck.

If nothing changes then it goes back to Windows.

I want to start Steam, play a game. No, Steam loads 20GB with shaders and I have to wait an hour for Steam to start.

https://pastebin.com/PK5uyYyA

[smintman]

Replying to https://github.com/ValveSoftware/steam-for-linux/issues/8076#issuecomment-994914000

Hi are you using experimental? Not 100% sure but I think every time an update to that is pushed (which depending on your settings could be quite often) it results in the shader cache data being downloaded again? Is that the issue?

[gardotd426]

Somebody please ban that guy. https://github.com/ValveSoftware/steam-for-linux/issues/8076#issuecomment-994914000

[jarrard]

Somebody please ban that guy. #8076 (comment)

Why? that is the incorrect way to handle Windows enthusiasts having frustrating issues under Linux!! By scorning frustrated newbies to Linux whenever they have a minor outburst, we're just telling everyone to go back to windows and let Linux gaming go away! Also these issues NEED to be resolved before Steam Deck is in peoples hands.

[jarrard]

For months he has said the problem will be fixed. No, even after 3 months it was not fixed.

EVERY time I start Steam, I have to download 20GB of shaders.

Make sure your not running games off NTFS partitions, if you are then be sure to symlink their compat and shadercache folders under SteamLibrary directories back to default USER .steam locations (never run those off ntfs).

Second of all I recommend exiting Steam, and deleting shadercache folders and reload steam. You should also disable the shadercache options in steam-settings before doing this just to be sure. Re-enable them afterwards.

This helped in my case a little bit, steam will download that initial bunch of cache again but afterwards should only download a little bit for each game. Its not a full fix bit it did appear to reduce the amount it was grabbing!

[gardotd426]

Somebody please ban that guy. #8076 (comment)

Why? that is the incorrect way to handle Windows enthusiasts having frustrating issues under Linux!! By scorning frustrated newbies to Linux whenever they have a minor outburst, we're just telling everyone to go back to windows and let Linux gaming go away! Also these issues NEED to be resolved before Steam Deck is in peoples hands.

That's not a minor outburst.

Literally calling kisak a liar and going on some ridiculous rant is not what this issue thread is for. Nor is it for discussion about such topics.

As kisak said, this system is relatively new. It improves by learning, and takes time. @Eisenvaddi provided zero actual relevant information (e.g. what games are repeatedly downloading GB+ precaching updates, literally anything else useful, etc).

The system will improve over time, and large GB+ downloads should become more infrequent as the system learns and becomes more effective. This is the nature of such a crowdsourced technology.

And as far as the Steam Deck goes, Steam Decks all have identical relevant hardware (CPU+GPU), the shadercache will be identical for every SD. Anyone that thinks SD shader precaching updates will work the same way that it works for people on regular Linux desktop is likely sorely mistaken. They only need to target one GPU for their shader caches, it will not be remotely like it is for regular desktop users.

[jarrard]

Literally calling kisak a liar and going on some ridiculous rant

Yes well that is a bit much, kisak is doing his/her best. BUT the rest of his comment isn't a ridiculous rant iyam... YOUR hostile response however... just chill, don't overreact! kiask doesn't need your protection! END OF DISCUSSION........

[gardotd426]

There's no overreaction, calling valve employees liars for no reason on a GITHUB REPORT THREAD is uncalled for.

This is not a discussion thread (again), this discussion is over.

[gardotd426]

@kisak-valve I'm being forced to reopen this issue. I've had five 2-3GB+ shader precaching updates for Borderlands 3 in the past 3 days. And it's back to downloading updates for every single game I have installed on almost every launch. Warframe is also getting GB+ updates.

It's to the point where I only even launch Steam if I know I don't need to be doing anything else remotely taxing for the next 30 minutes, and that I have to launch 30 minutes before I actually want to play any games, because otherwise performance will be horrible because of all the CPU usage.

I have a 5900X. Not some weak or even midrange CPU. This is kind of ridiculous.

[gardotd426]