geeksville commented 1 year ago

Reported by a kind redditor.

Behavior change coincided with a game version update: https://steamdb.info/app/570940/

geeksville commented 1 year ago

note to self:

possibly auto reflection is wrong - check these paths: https://steamdb.info/app/570940/ufs/ also make sure all of these default dirs are listed: https://partner.steamgames.com/doc/features/cloud

geeksville commented 1 year ago

Hi /u/squigs...,

okay! Can you do the following?

Start darksouls and do something which would normally cause a save game snapshot (so it will be captured in the steamback log files)
Exit dark souls
Soon after that - start desktop mode
Open a "Konsole" window - it will be in the list of apps on the left

At that UNIX shell prompt type (exactly):

find /home /run/media -name 570940 -exec find '{}' -type f ';' >darkfiles.txt
# this command lists all files in the dark soul app (including save files)

Attach two files to this bug (when you start typing a comment there is a button to 'attach files':
The darkfiles.txt file that is sitting in your home directory
The file at /home/deck/homebrew/logs/Steamback/plugin.log

You can probably do this most easily by opening a chrome window in steam desktop mode and browsing to github. Then you can directly access the files in your Steam deck home directoy.

If you have any questions about how to do this please add them to this bug and I'll help. After I have those two files I can probably fix this bug pretty soon (next few days). I'll ping you when I have an 'alpha' build or you can wait until the new plugin gets through the (slow) Decky Loader testing queue and it shows up in the 'store'. Installing an alpha build is a little tricky but I can provide instructions.

geeksville commented 1 year ago

(also please click to turn on 'notifications' on this github bug - so that you'll get an email when I have an update - I might need to ask for one or two more files TBD)

FScot commented 1 year ago

Here are the files geeksville! I really appreciate it.

Also, you don't necessarily need to send me an alpha, I'd just like to be able to replace the save with one of the last ones I was using yesterday if that would be easier. That way it would give you more time to incorporate this into a future build and not just for me.

darkfiles.txt plugin.log

geeksville commented 1 year ago

cool beans. the following are misc mostly unformatted notes from looking through the logs:

The new location for save games files seems to be:

/home/deck/.local/share/Steam/steamapps/compatdata/570940/pfx/drive_c/users/steamuser/Documents/NBGI/DARK SOULS REMASTERED/220460073/DRAKS0005.sl2
/home/deck/.local/share/Steam/steamapps/compatdata/570940/pfx/drive_c/users/steamuser/Documents/NBGI/DARK SOULS REMASTERED/220460073/steam_autocloud.vdf
/home/deck/.local/share/Steam/steamapps/compatdata/570940/pfx/drive_c/users/steamuser/Documents/NBGI/570940/220460073/steam_autocloud.vdf

A couple of days ago it seems like @fscot tried to restore from an undo file and that restore didn't work because wrong directory path moved to TBD location:

[2023-07-31 22:07:36,765][INFO]: Attempting restore of {'game_info': {'game_id': 570940, 'game_name': 'DARK SOULS™: REMASTERED', 'install_root': '/home/deck/.local/share/Steam', 'save_games_root': '/home/deck/.local/share/Steam/userdata/220460073/570940/remote'}, 'timestamp': 1690862789544, 'filename': 'undo_570940_1690862789544', 'is_undo': True}
[2023-07-31 22:07:36,767][INFO]: Copied 2 files from /home/deck/homebrew/data/Steamback/saves2/undo_570940_1690862789544 to /home/deck/.local/share/Steam/userdata/220460073/570940/remote

Unrelated exception in logfile:

[2023-08-01 17:15:19,850][DEBUG]: No rcf /home/deck/.local/share/Steam/userdata/220460073/653530/remotecache.vdf
[2023-08-01 17:15:19,850][ERROR]: Error scanning rcf for {'game_id': 746850, 'game_name': 'Cloudpunk', 'install_root': '/run/media/deck/e2fa5acd-626a-436e-a3d2-1a1ed0be3f2c/SteamLibrary'}, exception Traceback (most recent call last):
  File "/home/deck/homebrew/plugins/Steamback/py_modules/steamback/__init__.py", line 596, in try_rcf
    return self._read_rcf(info)
  File "/home/deck/homebrew/plugins/Steamback/py_modules/steamback/__init__.py", line 364, in _read_rcf
    saveRoot = self._find_save_games(game_info, rcf)
  File "/home/deck/homebrew/plugins/Steamback/py_modules/steamback/__init__.py", line 281, in _find_save_games
    d = self._search_likely_locations(game_info, rcf)
  File "/home/deck/homebrew/plugins/Steamback/py_modules/steamback/__init__.py", line 255, in _search_likely_locations
    addRoots(True)
  File "/home/deck/homebrew/plugins/Steamback/py_modules/steamback/__init__.py", line 241, in addRoots
    roots.append(self._get_game_saves_root(
  File "/home/deck/homebrew/plugins/Steamback/py_modules/steamback/__init__.py", line 166, in _get_game_saves_root
    installdir = self._parse_installdir(game_info)
  File "/home/deck/homebrew/plugins/Steamback/py_modules/steamback/__init__.py", line 128, in _parse_installdir
    vcf = _parse_vcf(os.path.join(
  File "/home/deck/homebrew/plugins/Steamback/py_modules/steamback/__init__.py", line 25, in _parse_vcf
    with open(path) as f:
FileNotFoundError: [Errno 2] No such file or directory: '/run/media/deck/e2fa5acd-626a-436e-a3d2-1a1ed0be3f2c/SteamLibrary/steamapps/appmanifest_746850.acf'

Hmm. For some reason it seems that new dark souls has no vcf files - just the two autocloud.vcf files. Check to see if multiple autocloud files are supported and make vcf scanning optional?

geeksville commented 1 year ago

btw - it will take me a couple of days to figure this out, but if it would be useful tomorrow I can give you a couple of command line params that will let you (probably) restore your old/existing two week old dark soul snapshots into the current dark souls install. Would that be useful?

FScot commented 1 year ago

I have the saves from 2 weeks ago, I want to restore the saves from 2 days ago. Any thoughts on how I can get to those?

geeksville commented 1 year ago

alas nope - since steamback wasn't getting notified of the changes (due to this move and steamback not finding them) it hasn't been backing up dark souls since their update.

geeksville commented 1 year ago

misc notes: continuing investigation. remotecache file is here: /home/deck/.local/share/Steam/userdata/220460073/570940/remotecache.vdf

two days ago a scan of dark souls did not find any changes (presumably because we are searching the wrong directory): [2023-07-31 22:07:11,380][INFO]: Attempting backup of {'game_id': 570940, 'game_name': 'DARK SOULS™: REMASTERED', 'install_root': '/home/deck/.local/share/Steam'} [2023-07-31 22:07:11,380][DEBUG]: RCF is valid /home/deck/.local/share/Steam/userdata/220460073/570940/remote [2023-07-31 22:07:11,381][ERROR]: Error reading JSON for save_1245620_1680055802810.json, Expecting value: line 1 column 1 (char 0) [2023-07-31 22:07:11,381][WARNING]: Skipping backup for 570940 - no changed files

TODO Fix (unrelated apparently) bug of never deletinging corrupted json files.

hmm - need to see contents of the vdf file and autocloud files to know more...

@fscot Thanks for the logs! I think I should be able to figure out how to fix steamback so it works with the new darksouls (and other games that might make similar changes). But I will need five more files from your machine to figure out what's up. Would you mind adding them to this bug? (note that four of these files have the same name - just different directories) The files are:

/home/deck/.local/share/Steam/userdata/220460073/570940/remotecache.vdf
/home/deck/.local/share/Steam/steamapps/compatdata/570940/pfx/drive_c/users/steamuser/Documents/NBGI/570940/220460073/steam_autocloud.vdf
/home/deck/.local/share/Steam/steamapps/compatdata/570940/pfx/drive_c/users/steamuser/Documents/NBGI/DARK SOULS REMASTERED/220460073/steam_autocloud.vdf
/home/deck/.local/share/Steam/steamapps/compatdata/570940/pfx/drive_c/users/steamuser/Documents/NBGI/570940/220460073/steam_autocloud.vdf
/home/deck/.local/share/Steam/steamapps/compatdata/570940/pfx/drive_c/users/steamuser/Documents/FromSoftware/DARK SOULS REMASTERED/220460073/steam_autocloud.vdf

Note: these files are in the .local subdirectory of the deck user account. I think that directory is usually hidden. If you are using the file browser to find them you'll probably need to click some option to 'show hidden files'. Sorry - I'm away from my steamdeck at the moment (and usually I use ssh anyways - which is probably not something you want to deal with) so I'm not sure of the exact location in the GUI.

Also - if you want a temporary workaround until steamback is fixed: you could run the following command from desktop mode any time you want to 'backup' your darksouls save:

cp /home/deck/.local/share/Steam/steamapps/compatdata/570940/pfx/drive_c/users/steamuser/Documents/NBGI/DARK SOULS REMASTERED/220460073/DRAKS0005.sl2 ~/darkbackup.sl2

Then to 'restore' you would reverse the order of those two parameters (but be careful)

geeksville commented 1 year ago

@FScot ?

FScot commented 1 year ago

Hey, geeksville! Sorry I've been super busy and haven't been able to respond. I will try these things today and get back to you.

geeksville commented 1 year ago

no worries - thanks!

FScot commented 1 year ago

Hey geeksville! Sorry for the delay. So here are the files, except the second in your list is the same as the 4th file.zip

geeksville commented 1 year ago

Thanks! Alas I ended up unexpectedly in hospital (bummer). So it will be a few days before I'll be able to GitHub.

(Sent from a phone - please ignore typos)

On Sun, Aug 6, 2023, 11:25 FScot @.***> wrote:

Hey geeksville! Sorry for the delay. So here are the files, except the second in your list is the same as the 4th file.zip https://github.com/geeksville/steamback/files/12268587/file.zip

— Reply to this email directly, view it on GitHub https://github.com/geeksville/steamback/issues/38#issuecomment-1666682320, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABXB2J66HVOBE7JDB3ESA3XT36ALANCNFSM6AAAAAA3ANBRGU . You are receiving this because you authored the thread.Message ID: @.***>

FScot commented 1 year ago

Holy smokes! Take care!

geeksville commented 1 year ago

ok - home now but it will be a few days before i'm coding again. Probably not till mid next week.

FScot commented 1 year ago

NP, take it easy man

geeksville commented 1 year ago

thanks. I'm back programming now and should be able to hopefully wrap this one up this weekend.

geeksville commented 1 year ago

misc notes as I investigate:

remotecache file very intersting

"570940"
{
    "ChangeNumber"      "-1801319871458445243"
    "ostype"        "0"
    "NBGI/DARK SOULS REMASTERED/220460073/DRAKS0005.sl2"
    {
        "root"      "2"
        "size"      "4326608"
        "localtime"     "1691273379"
        "time"      "1691273379"
        "remotetime"        "1691273379"
        "sha"       "f16b9d183441b8732eea8ff1b2713da955671152"
        "syncstate"     "1"
        "persiststate"      "0"
        "platformstosync2"      "-1"
    }
    "A_WD_RANKING_VID_BOUNTY_76561198180725801"
    {
        "root"      "0"
        "size"      "1076"
        "localtime"     "1689652017"
        "time"      "1689652016"
        "remotetime"        "1689652016"
        "sha"       "f6803dafc1ea7e5520e646113da8dbb618c22ea0"
        "syncstate"     "1"
        "persiststate"      "0"
        "platformstosync2"      "-1"
    }
    "B_WD_RANKING_VID_BOUNTY_76561198180725801"
    {
        "root"      "0"
        "size"      "1076"
        "localtime"     "1689652020"
        "time"      "1689652019"
        "remotetime"        "1689652019"
        "sha"       "20c1df0a70d0691a52eb156b31c9e22253b8feea"
        "syncstate"     "1"
        "persiststate"      "0"
        "platformstosync2"      "-1"
    }
}

all autocloud files are just:

"steam_autocloud.vdf"
{
    "accountid"     "<number>"
}

problem 1

The remotecache mentions A_WD_RANKING_VID_BOUNTY_76561198180725801 and that file exists in the 'new style' steam 'remote' directory. So it is found first in _find_save_games and that shortcircuits/prevents scanning to find the autoclouds.

/home/deck/.local/share/Steam/userdata/220460073/570940/remotecache.vdf
/home/deck/.local/share/Steam/userdata/220460073/570940/remote/B_WD_RANKING_VID_BOUNTY_76561198180725801
/home/deck/.local/share/Steam/userdata/220460073/570940/remote/A_WD_RANKING_VID_BOUNTY_76561198180725801

problem 2: this project has multiple autocloud directories

I bet multiple autocloud files are causing us to find the wrong root for NBGI/DARK SOULS REMASTERED/220460073/DRAKS0005.sl2. We'll need to add support for that I guess...

TODO

support multiple autocloud/remote directories.
keep each backup file info as a 1:1 relation with the filename mentioned in remotecache and the directory where we found that particular file
store that full relative path as a new bit of info in the json file
if that info is not found in the json file at restore time fallback to rescanning all directories to find the correct dir (to let old json files still work)

geeksville commented 1 year ago

Hi @fscot,

Okay - I found the problem. Alas, I'm still stuck with my kinda crummy home laptop so I'm a bit scared to make the big refactor/testing a few other games until back at my main dev machine. So I'm sorry but I won't be able to actually fix this one until early September (I get back to work sept 1).

Mainly I need to be super careful to not break the 99% of existing games in fixing what this one game revealed.

FScot commented 1 year ago

No worries! Do what you need to do!

geeksville commented 1 year ago

woot - I can now reproduce the problem here. It also occurs in the new release of No Man's Sky (which I own). I think the Steam developer SDK changed to introduce this new feature and it is now showing up in new game releases. The fix is straightforward but I want to test it carefully (because it will affect all games - including the 95% games which were still working with the old code).

geeksville commented 1 year ago

Hi @FScot,

Good news! I think based on your logs I have a good fix for your game (and others). Would you mind trying it? If it works for you I'll release it more broadly.

Steps to install (since it is not yet in the store):

Download Steamback.zip to somewhere on your steamdeck (you can do this from chrome). The file is located here: https://github.com/geeksville/steamback/releases/tag/1.0.0
In Decky loader uninstall your existing Steamback (don't worry any existing saves will not be affected).
In the Decky Loader menus go to gear / Developer and then choose "Install Plugin from ZIP file". Using the provided selector, point decky at the zip you downloaded.
(important) Reboot your steam deck (Decky loader currently has a bug wrt reloading plugins).
Try dark souls and see if backup/restore works again!

I'm super eager to hear how it goes.

geeksville commented 11 months ago

@FScot This version is now up in the decky "testing" store. Can you change your decky settings to use that store? Then you should be able to update to 1.0.0 of Steamback.

If you could confirm/deny it works for you that would really help a lot!

geeksville commented 6 months ago

closed due to non response. If still a problem please reopen.

geeksville / steamback

"Dark Souls" game stopped working with steamback about two weeks ago #38

problem 1

problem 2: this project has multiple autocloud directories

TODO