CryoByte33 / steam-deck-utilities

A utility to improve performance and help manage storage on Steam Deck.
GNU General Public License v3.0
3.23k stars 81 forks source link

[Bug] The storage management tool can in some cases prevent shader cache generation and prevent game data from being moved. #79

Open sevansup opened 1 year ago

sevansup commented 1 year ago

The TLDR is that if broken symbolic links are present when using this tool, then: Games will A) not be detected by the tool, and B) shader cache generation can stop working for that game. This can seemingly only happen if you manually delete the compatdata or shader cache folders for the game, but the symbolic link remains (in my testing, this happened specifically with the game's broken shader cache symbolic link remaining, I am only assuming the same may apply to compatdata).

This could be more likely to occur in the event that you're an older Deck user like me (I've nearly had mine for a year). At some point, Valve may have stored compatdata for a game on the drive the game was installed on. Some of my oldest installed games have their compatdata folders on the SD card, for example, whereas anything more recent has the data on the SSD only, regardless of which drive the game is installed on. It's also possible that this compatdata was moved when using the 'move game' feature within Steam, or as a bug, and was maybe never supposed to end up there.

In any case, this may be unlikely to occur for future users but it happened to me and it's how I found this bug. I was trying to go in and clean out my old compatdata folders on my SD card. I had them for about 12 games. Most of these games were not detected by the storage tool in CryoUtilities, which is fine--this data probably shouldn't have ended up here anyways. Some of these games, like the Valheim example below, had compatdata BOTH on the SD card and on the SSD - possibly a bug when moving the games via Steam as I mentioned. These were detected by the tool.

I'll use the Valheim example (Steam ID: 892970). I used this new feature in CryoUtilities to move the game data from my SSD to my SD card. However, due to the above reasons, I also had compatdata on my SD card for this game, and wanted to clean that up, but wasn't sure which directory was the "main" one. So I uninstalled the game, then manually deleted all compatdata folders for it, including the ones in the new cryoutilities folder on the SD card. I did the same for the shader cache, though I forgot to then go and delete the symbolic link cryoutilities had created for the shader cache.

I then reinstalled the game to my SD card. Compatdata was generated on the SSD as expected. I loaded up the game to generate a shader cache. Then closed it. Then opened CryoUtilities, tried to move the game data, and nothing was detected. Eventually I discovered that a leftover shader cache symbolic link was present, and this was preventing the shader cache from being generated for the game. It was also preventing it from being detected by the tool.

I deleted that symbolic link, opened the game, a proper shader cache folder was generated, then I was able to close the game and successfully move it via CryoUtilities.

So perhaps this is more of a user error than anything else, but I do think it will happen again to someone and perhaps a detection method for broken symbolic links could be added so that this bug doesn't occur, as having no shader cache for a game could definitely result in some performance problems.

sevansup commented 1 year ago

Perhaps more concerningly, I am now noticing another issue.

Another one of those older games I installed long ago, for which compatdata mysteriously existed on my SD card, is River City Girls (see my post above for explanation on this weird phenomenon).

I repeated the steps above for it that I did for Valheim, ensuring no leftover symbolic links were present, no compatdata or shader cache folders. All data should be gone for the game, right?

However, upon reinstalling River City Girls to the SD card, CryoUtilities still would not detect the game data on my SSD and move it over. I tried going into Steam and using the "move game" function to transfer the game over to my SSD. Then I opened CryoUtilities, and sure enough -- the game is detected. I moved it back to my SD card via Steam, and it was still detected properly.

I further tested with another of these titles -- Cook, Serve, Delicious. On this one, no matter what I do--uninstall the game, obliterate compatdata folders, move it back and forth within Steam to SD and SSD, CryoUtilities will not move the game data any further. It did it once to the SD card, but it won't go back the other way. If I uninstall the game, CryoUtilities still detects the game data for this app as "installed." Something fishy is going on, and I may re-image my deck (though I haven't done any tweaks to it besides CryoUtilities and EmuDeck)...

So far this only happens with these "phenomenon" games that had the mysterious compatdata folders already on my SD card, and in some cases BOTH on the SD and SSD. Anything I've installed in recent months has ALWAYS put the compatdata on the SSD. So I'm just not sure what happened here.

This whole thing leads me to believe that somewhere in Steam there is a possibility that, perhaps when moving a game and having the transfer process fail (or for other unknown reasons like an old method of storing compatdata, or the way it handles compatdata when the drive is nearly full) Steam (and by extension CryoUtilities) can think that your game is located where it isn't, and it won't be detected in certain situations. This even persists when uninstalling the game and removing the compatdata folders. My first hunch is maybe the "userdata" folder stores something related to this but I don't know.

It is certainly bizarre.

CryoByte33 commented 1 year ago

Hey, I'm not able to give a full response now, but the issue is very likely that the Steam cache VDF only refreshes when Steam starts, hence my recommendation to get into and out of game mode, or reboot, if you notice weird things.

Games won't appear properly because of it, and there's not much I can do unless Steam starts updating the cache files while running.

As for the old data, I'll have to think about how I could handle that without persisting state. I fing think that there's a lot I can do other than assume you're okay with deletion and removing things, which would be lossy 🤔

sevansup commented 1 year ago

Thank you for your response! I love your work and appreciate what you've done so far. I missed the bit about restarting Steam and indeed that seems to alleviate some of the weirdness I'm experiencing, and I can get data to consistently transfer now.

However, with further testing I can reproduce certain games are installing their compatdata to /run/media/mmcblk0p1/steamapps/compatdata/ where they aren't picked up properly by CryoUtilities initially. I am not sure what has caused this, and it only seems to apply to games I had installed a long time ago, but persists no matter how many times I restart the Deck/Steam or uninstall/reinstall these games.

If I'm the only one seeing this, I'll consider re-imaging my deck. I can reproduce this 100% of the time by following these steps: Install the "cursed" game to the SSD. Right click on the game in Steam > Properties > Local Files > Move Install Folder > transfer it to my SD Card. Instantly, a compatdata is created in the aforementioned folder on my SD card. The old compatdata folder remains in /home/deck/.local/share/Steam/steamapps/compatdata/ so I'll have 2 copies of it.

I do think the symbolic link leftovers is a separate issue and could maybe trip up others as it did me, but if someone is tech savvy enough to know about compatdata folders and manually delete them, they can probably make sure to clean up their symbolic links, so perhaps it isn't a big deal. Thanks for your input!


Edit: Looks like I'm not entirely alone, other users have reported these inconsistencies with the compatdata folders sometimes being on the SD card, or both the SD card and the SSD: https://github.com/ValveSoftware/steam-for-linux/issues/5717#issuecomment-1171524282

Perhaps symbolic linking the /run/media/mmcblk0p1/steamapps/compatdata/ folder to > /home/deck/.local/share/Steam/steamapps/compatdata/ would fix this issue. didn't work

Edit 2: Tested on a freshly reformatted Steam Deck, completely default everything. Indeed, I can definitely say these compatdata folders on the SD card are created when moving a game over from the SSD to the SD card. If you install a game on the SD card first, they are only created on the SSD. Likewise, if you move a game from the SD to the SSD, they are not created on the SD card. Duplicates only happen when moving from SSD to SD via Steam with no apparent way to fix it. Both directories appear to be used in this case.