Steam Cloud path conflicts breaks game-save syncing

[da2x]

TL;DR: Paths are broken because of XDG overwrites. Skip to the code example.

Update: Read this analysis of this bug which includes two suggestions for possible fixes. The below ended up serving as drafts for that analysis and is less less accurate.

---

For most games, Steam Cloud syncs saves and configurations to and from /home/user/.local/share/studio/gamename. This is the LinuxXdgDataHome data root path in Steam Cloud, which documentation suggests is corresponding to $XDG_DATA_HOME. It appears that game developers use $XDG_DATA_HOME (default in e.g. Unity3D) but Steam itself doesn’t actually read the $XDG_DATA_HOME variable but just hardcodes $HOME/.local/share/ or $HOME/.config/.

---

Many games, like RimWorld , won’t read from /home/user/.local/share/studio/gamename where saves are synced by Steam but instead only read from and write to /home/user/.var/app/com.valvesoftware.Steam/data/studio/gamename. These won’t be synced and players won’t have access to existing saves.

Some games, like Cities Skylines, are synced to /home/user/.local/share/studio/gamename and will read from both locations, but only write changes and new saves to /home/user/.var/app/com.valvesoftware.Steam/data/studio/gamename. Players have access to existing syncs, but can’t update existing saves or add new ones.

A few games, like FTL, are synced to /home/user/.var/app/com.valvesoftware.Steam/data/studio/gamename and everything appears to work out of the box. These games seem to be the exception.

---

I tested with restoring the sandboxed $HOME prefix stuff in the wrapper committed by @alexlarsson and reverted by @kujeger. This seems to fix cloud syncing with many games, but broke some games like Cities Skylines.

Combining this special sandboxed $HOME variable with having $XDG_DATA_HOME point to /home/user/.var/app/com.valvesoftware.Steam/home/.local/share/ and $XDG_CONFIG_HOME point to /home/user/.var/app/com.valvesoftware.Steam/home/.config/ should fix pretty much every game as Steam and games then should agree on where saves are kept. Should be a workable solution that works for a majority of games (at least Unity3D titles, and I suggest most others too) — unless anyone has contacts at Valve and can make them fix this upstream?

Here is a suggested patch for `steam-wrapper:

export HOME="$HOME/.var/app/com.valvesoftware.Steam/data/home"
export XDG_CONFIG_HOME="$HOME/.config"
export XDG_DATA_HOME="$HOME/.local/share"

OLD_DATA_DIR=$HOME/.var/app/com.valvesoftware.Steam/data/Steam
if [ -d $OLD_DATA_DIR ]; then
    mkdir -p $HOME
    mkdir -p $XDG_CONFIG_HOME
    mkdir -p $XDG_DATA_HOME
    rm -rf $OLD_DATA_DIR
fi

The good:

• All games† now only have one data directory and everyone agrees on where that directory is located.
• All games† now sync to and from Steam Cloud, meaning players get back years worth of game progression.

The bad:

• The old Steam library and install is no good. It can’t be migrated as moving it breaks some games that use absolute install paths and causes crashes with the Steam overlay.
• Local saves inside and outside the Flatpak container is lost. These are recoverable but we can’t migrate the data for the user without causing Steam Cloud conflicts. We also don’t know what the actual paths that need to be migrated are. Some are inside the Flatpak app directory, some are in the home directory, some are partially in one place and partially in the other.

† Works with at least RimWorld, City Skylines, Borderlands 2, Project Highrise, Sim Airport (game doesn’t work, but sync does), Transport Fever (game doesn’t work, but sync does), Mini Metro, Dream Daddy, and Space Run (game doesn’t work, but sync does).

The following titles have broken cloud sync implementations and can’t be fixed without creative game-specific patching that has nothing to do with Flatpak:

• FTL (casesensitive path mixup)
• Bridge Constructor Portal (cloud profile doesn’t match actual storage location)

[nanonyme]

It sounds to me like this thing is so overly messed up there should be a counterpart ticket on Steam for Linux project

[da2x]

It sounds to me like this thing is so overly messed up there should be a counterpart ticket on Steam for Linux project

Valve’s only fault here is that they make a poor recommendation in their documentation. I’ve contacted them about that specifically.

[da2x]

I wrote up a more thorough analysis of this bug on my blog. It’s way less rambly than the above.

[kujeger]

Just to clarify: #22 does not give steam access to the actual host $HOME -- it simply persists everything written to $HOME inside the steam flatpak.

(the option to give access to host $HOME is --filesystem=home)

Great job describing this mess, I never discovered these issues when testing.

It seems that simply setting $XDG_DATA_HOME to $HOME/.local/share (along with the other XDG*s) would fix the issue, but would also lose the progress of any applications actually honoring the XDG paths, unless manually moved, as you point out.

[da2x]

[…] Just to clarify: #22 does not give steam access to the actual host $HOME -- it simply persists everything written to $HOME inside the steam flatpak.

I don’t think this can be solved in any other way than providing documentation. Maybe xdg-open "https://github.com/flathub/com.valvesoftware.Steam/wiki" on upgrade or steam-wrapper first-run to a page describing where to find game saves and where to move them to recover data.

[kujeger]

https://support.steampowered.com/kb_article.php?ref=6736-QEIG-8941#conflicts

It looks like steam will detect and require manual intervention if there are sync conflicts, so it's possible that we can just copy everything from the old XDG_DATA_HOME to a new one.

If so the wrapper could just more or less be

cp -a $XDG_DATA_HOME/* $HOME/.local/share/
export XDG_DATA_HOME=$HOME/.local/share/

and similar for cache and config, with something to make sure that it only runs once. There's no need to change $HOME.

[swick]

Steam installs itself into $XDG_DATA_HOME so the script would also copy steam and all games.

[kujeger]

Then perhaps moving $XDG_DATA_HOME/Steam, and copying everything else.

[da2x]

You can’t move $XDG_DATA_HOME/Steam. Some paths in that directory itself are hardcoded.

[alexlarsson]

You can move it and leave around a relative symlink to the new place.

[kujeger]

That's a good thought; instead of changing the env variables we could

• move/copy $HOME/.local/share/* to $XDG_DATA_HOME/
• (relatively) symlink $HOME/.local/share to $XDG_DATA_HOME

[alexlarsson]

Same with .config

[alexlarsson]

The blog about this mentioned that some apps did not handle symlinks well (even crashing). Should get some data about that first though.

[da2x]

The problem with moving or symlinking anything is that there are already two directories per game. One created by the game executable ($XDG_CONFIG_HOME/game), and one by Steam Auto-Cloud ($HOME/.config/game). Which of the two directories do the user want to keep? I’d personally want my cloud saves back; there are a lot of hours in there — but others may want to keep their more recent game saves instead.

These games crash with symlinked data directories: Oil Rush and Colony Survival.

[nanonyme]

Any reasonable way to query the choice from end-user?

[TingPing]

If you symlink $HOME/.config to $XDG_CONFIG_HOME the broken games would be fine and steam would use the right directory?

[da2x]

We need to make a decision on what to break to proceed with this issue. Local saves or cloud saves.

I’ve experimented a bit with merging local folders on top of steam cloud folders. A dialog in Steam will ask players to choose which they want to keep. However, we shouldn’t overwrite either the local copy of the Steam Cloud or the local game progress to make it possible to recover either two.

I believe this should fix the issues and preserve both sets of data:

1. Create ~/.var/app/com.valvesoftware.Steam/home/ and set it as $HOME in the launcher.
2. Create ~/.var/app/com.valvesoftware.Steam/home/.cache/ and set it as $XDG_CACHE_HOME in the launcher.
3. Create ~/.var/app/com.valvesoftware.Steam/home/.config/ and set it as $XDG_CONFIG_HOME in the launcher.
4. Create ~/.var/app/com.valvesoftware.Steam/home/.local/share/ and set it as $XDG_DATA_HOME in the launcher.
5. Copy ~/.var/app/com.valvesoftware.Steam/config/* to ~/.var/app/com.valvesoftware.Steam/home/.config/
6. Move ~/.var/app/com.valvesoftware.Steam/data/Steam/ to ~/.var/app/com.valvesoftware.Steam/home/.local/share/Steam/
7. Copy ~/.var/app/com.valvesoftware.Steam/data/* to ~/.var/app/com.valvesoftware.Steam/home/.local/share/

The Steam library folder shouldn’t be copied as it’s where all games is installed. It can be enormous. The cache folder doesn’t need to be copied.

[TingPing]

@da2x If that works it sounds fine.

[nanonyme]

I'd kinda lean towards backupping local saves, replacing with cloud saves where available and alert user they need to manually override cloud data with local data if they didn't like the choice. It's harder to recover from the point if you break the user's cloud data

[AsciiWolf]

Any way to properly fix this?

[nanonyme]

@AsciiWolf possibly although complex to fix. @TingPing has a branch with WiP patch in case you want to contribute

[gasinvein]

@da2x why symlink $XDG_DATA_HOME to hardcoded default ~/.local/share/, and not the opposite?

[nanonyme]

Just a note here: symlinks in the direction as in the patches (for at least XDG_CONFIG_HOME) break the Flatpak's ability to start even bash. It's basically a no-go

[nanonyme]

I managed to now get the migration logic started by @TingPing working as part of #110 I'm leaving the PR open for a bit in case you guys want to discuss but it should be mostly good to merge at this point

[nanonyme]

I recommend building from wip branch yourself if you want to try it out http://docs.flatpak.org/en/latest/first-build.html is sufficient in helping you to do a local build for testing Note that the migration will permanently change through symlinks (non-overridden Flatpak values) $XDG_CONFIG_HOME -> ~/.var/app/com.valvesoftware.Steam/.config $XDG_DATA_HOME -> ~/.var/app/com.valvesoftware.Steam/.local/share $XDG_CACHE_HOME -> ~/.var/app/com.valvesoftware.Steam/.cache ALERT: the migration process merges directories so obtaining the original directory structure through even manual rollbacking is most likely really hard. You can take backups before migration if you like. Note that $XDG_DATA_HOME/Steam is massive, keeping a copy of it might not be practical This is mostly about trying to find games that crash with the new structure.

[nanonyme]

There's a backup plan of trying bindfs if symlinks done this way still causes crashing but if there's no issues with this approach, we can adopt this. Imo if further changes are done, the bootstrapper should be turned into a Python script that execs Steam since the migrating logic is getting quite involved. We may also potentially need to start versioning migrations in case we need to migrate again in the future

[TingPing]

So I did very basic testing of this:

• Migrated existing data
• Made a cloud save in Civ V
• Restart app and load cloud save

Not sure if that is good enough but it worked at least.

[da2x]

Its import to test what happens with games that have a local game save before the migration, and also have an existing cloud save that will start syncing after the migration. Steam should ask you to resolve sync conflicts. You should not loose either the local or cloud saves without Steam t least asking you to choose. (I don't have any time to look into this for at least a few weeks.)

[nanonyme]

That's probably fine. It's not as if this issue is likely to go anywhere in the meantime. It'd be really nice to get also some more information on those crashes to understand better why they happened.

[michel-slm]

@nanonyme @TingPing @da2x - are there test scenarios you'd like help with? (If insufficient testing is what's preventing the test flatpak from being committed).

[nanonyme]

@michel-slm we basically just need that migration scenario described by @da2x tested with as many games as possible. Rollback is basically impossible if we get crashes

[nanonyme]

@da2x do you happen to by the way know any sample games that currently save to "wrong" place? I'm not totally sure if I have any in my catalog.

[da2x]

@nanonyme, there is a short list of popular titles with the issue confirmed near the end of the initial post.

Statistically, almost every game built on Unity 3D engine version 4 or newer will use $XDG_CONFIG_HOME in-game and then use $HOME/.config in their Steam Cloud configuration. Look at 2013 or newer here in this list: https://en.wikipedia.org/wiki/List_of_Unity_games

[da2x]

Thanks to GDPR you can inspect your Steam Cloud including seeing the time stamp for when files were last modified or uploaded via the web now: https://store.steampowered.com/account/remotestorage

Should help with testing and verifying that things are uploaded. Thanks European Parliament and the Council of the European Union!

[nanonyme]

Looks like I have exactly one game on the list and I don't have enough disk space to install it currently. I just built a new bundle to same location since wip branch was rebased to have the new Steam version too

[nanonyme]

Tested with Wasteland 2 that this results in Steam to upload data as exacted. Haven't tested a conflict scenario yet

1) Install old app 2) Run game 3) Do a save 4) Update to new app

Errors when launching Wasteland 2 disappeared as a result

[nanonyme]

Test round 2 with Wasteland 2 with existing cloud data was a success

1) Wiped entire .var/app/com.valvesoftware.Steam 2) Installed new app 3) Ran game again 4) Continuing the previous game that was previously uploaded to cloud

[nanonyme]

Test round 3 with both local and cloud data failed

Conclusion is that the migration script wipes local data on name conflicts so on eg Wasteland 2 you lose all local progress without any promptms. No go, we need to improve the config directory migration

[nanonyme]

Still haven't figured out a way to get Steam to trigger cloud sync conflict page so user could choose which stuff to use

[da2x]

If I remember correctly, then conflicts dialog is triggered when file on disk is newer than cloud copy, at which point Steam compares checksums, and asks users whether to override the local file or not.

This is documented in detail somewhere in the documentation for Steam Cloud on Steamworks.

[nanonyme]

Yeah, I haven't managed to trigger it here yet. Also I have a suspicion currently that preserving your Steam data dir breaks the test setup since cloud sync seemed to keep working even without migration

[nanonyme]

Tested latest revision of the migrate script Local data was preserved and uploaded into cloud. However I'm a bit concerned I still haven't seen the conflict dialog ever during my tests @da2x

[nanonyme]

Closing as fixed, sync seems to now work

[michel-slm]

Glad the issue is solved, but for reference the game I tested with (that works using the test Steam flatpak but broke with the earlier published flatpak) is Invisible Inc.