search

DFHack / dfhack

Memory hacking library for Dwarf Fortress and a set of tools that use it
Other
1.86k stars 465 forks source link

Manage DF data in the cloud #3969

Open myk002 opened 10 months ago

myk002 commented 10 months ago

As a prerequisite for Cloud Gaming (https://github.com/DFHack/dfhack/issues/3574), DFHack would need to provide a mechanism for players to copy their DF savegame and config data to an alternate directory that is managed by DFHack's Steam Cloud. This is necessary because DF doesn't offer Cloud saving by itself.

We could either add a separate top-level directory (e.g. dfhack-cloud/) for cloud backup or use a subdirectory under dfhack-config/, which is already enabled for Steam Cloud.

To save/restore DF game settings, we would need to act before DF is started, which means implementing at least some functionality in dflaunch (or the dfhack script? what platform does cloud gaming run as?).

There could also be a plugin, which would provide an in-game enablement and management interface.

when the functionality is enabled, either the plugin or the launcher would rsync data from save/ to the cloud save directory whenever DF saves the game. before game load, the launcher (must be the launcher since a plugin wouldn't be able to do it early enough) would rsync data from the cloud save directory to the DF directories.

dcraig327 commented 10 months ago

Cloud gaming on GeForce NOW runs as Windows 11.

Envision a server with only one game installed, the game you choose to play, in a Windows 11 VM and the first thing you see is the familiar Steam App showing the Library with only that game purchased and loading in the background.

dcraig327 commented 9 months ago

Personally I like dfhack-cloud/. So much so to rename dfhack-config/ to dfhack-cloud/ and adding the saves to dfhack-cloud/save/. This still allows the base game to enable Steam Cloud functionality in the future, also it's easy to understand for users and adds minimal clutter to the root directory. Yet, because I like it doesn't mean it's necessary or even worthwhile. In fact whatever location you choose to store the save games & config data will discovered and documented for all users to see on the PC Gaming Wiki page: https://www.pcgamingwiki.com/wiki/Dwarf_Fortress

Steam does the Cloud Sync right before the game is launched and also immediately after the game exits. This dfhack-cloud/save/ directory would be sync'd upon entering launchdf.exe and however you've hooked dfhack to be be notified right before the DF process terminates, as long as the saves are in this dfhack-cloud/save/ before that happens, then it'll be uploaded to the cloud.

Yes, only sync to/from the cloud save directory when the user has, you may have worded this differently, enabled cloud functionality. Within Steam as you can disable it two ways, under the games properties and in Steam Settings > Cloud.

lethosor commented 9 months ago

I want to avoid renaming dfhack-config to dfhack-cloud. It's not appropriate for non-cloud players, who would likely be the majority for a while.

More generally, I think a plugin would be ok. I think it would need to be optional in order to (a) avoid unwanted bandwidth consumption and (b) avoid consuming Steam cloud storage (is there a limit? I thought that was why DF opted not to enable Steam Cloud for everyone...)

myk002 commented 9 months ago

There are technical size limits, but they are reasonably large (though it's still very possible for a player to exceed the 10GB size limit). My understanding about why DF does not sync savegames to the cloud, though, is that Toady didn't want the players to experience the sync delay.

Edit: the number of files, though, is a concern. My save directory right now is only 1.4G but contains >18,000 files. We'd likely have to zip the directories just to reduce the file count if we were to sync the save dir to the cloud.

image

I opened a support request to Steam asking for more information about what would be possible:

I have a question about how my app might function with Geforce NOW. DFHack (app id: 2346660) runs as an in-process addon to Dwarf Fortress (app id: 975370). DFHack and Dwarf Fortress install to the same directory, and DFHack is dynamically loaded by Dwarf Fortress if the files are present. When DFHack is run from the Steam client, our launcher process runs the Dwarf Fortress app.

If we make DFHack available to Geforce NOW players, can our launcher dynamically install Dwarf Fortress on the Geforce NOW VM if the player has purchased Dwarf Fortress?

They said:

Sorry, but you'll need to reach out to nVidia about this. Our support for GeForce NOW just resolves around us allowing apps to participate, but we're not the experts on how things really work on that platform.

I then wrote an email to the Geforce NOW team:

I have an app published on Steam: DFHack (app id: 2346660) https://steamdb.info/app/2346660/

It runs as an addon to Dwarf Fortress (app id: 975370) https://steamdb.info/app/975370/

DFHack and Dwarf Fortress install to the same directory, and DFHack is dynamically loaded by Dwarf Fortress if the binary files are present. When DFHack is run from the Steam client, the launcher process runs the Dwarf Fortress binary.

If we make DFHack available to Geforce NOW players, can our launcher dynamically install Dwarf Fortress on the Geforce NOW VM via the Steam API if the player has purchased Dwarf Fortress?

Also, does Dwarf Fortress have to explicitly support GeForce NOW itself, or can my app proxy that support, e.g. by managing cloud saves on behalf of the Dwarf Fortress app?

but I haven't seen a response yet.

dcraig327 commented 9 months ago

I imagine that nVidia GFN doesn't allow Installing 2 games with different appid's (DF & dfhack). Technically they are the same game as they both install to the same directory, but within the Steam client it requires installing two different games. I do feel they should add this feature and your wording to nVidia is great.

Just tested SKSE (Skyrim Script Extender) as I own that and the original Skyrim on Steam. And after installing only SKSE and after clicking Play, I'm prompted to install Skyrim as if I had clicked the Install button on Skyrim. This could be mentioned to nVidia if they're stumped. It also could be added to dfhack. I did check SKSE and there's no mention of the Skyrim depot at all: https://steamdb.info/app/365720/depots/

I mentioned X3: Albion Prelude as an example of a "DLC" that installs the base game, but this DLC isn't mentioned in the base game's store page. I just checked that games depot and it has the entire listing for the base game X3: Terran Conflict there and the size of that depot matches the size of the base game's depot: https://steamdb.info/app/201310/depots/ https://steamdb.info/app/2820/depots/

Regarding having to zip up the save folder to speed up cloud syncing. I imagine a low priority process could be created in the background handling that while the game is running. Low priority. Also, possibly compressing the majority of files that are least likely to change and flagging that archive dirty if one of those files do change. To me that sounds like a lot of work, setting up a database for each file almost. Yet I imagine if anyone really was worried about minimizing the overhead time of syncing saves to and from the cloud, better ideas would come. Logging a count of how many times a file was written to during a playthrough, just a file name and a number. That'd be an efficient way of creating a starting point of how the files would be organized in these archives. Having multiple archives would be preferred.

Compression would happen in the background on one thread assigned a lower priority while the game is running. Yet when the game isn't running. You can create multiple threads to use almost all of the CPU's capabilities and speed up the decompression and compression.

zip likely isn't the most efficient. 7z I imagine is the fastest. Found an open source GPL3 project that indicates it's based off of 7z and much faster: https://peazip.github.io/fastest-compression-zip-deflate-benchmark.html

dcraig327 commented 9 months ago

Source to that old version of SKSE listed in Steam: https://github.com/NightQuest/SKSE/

dcraig327 commented 9 months ago

but I haven't seen a response yet.

I have a feeling it may remain this way. To me a a lack of a response past 24 hours means I get to get louder until there is a response. It could be some corporate excuse mentioning security, but a lack of a response is unprofessional.

Control what you can control.

I say ignore Cloud Gaming at the moment as this seems to be held up by means outside of your control.

I say add DF saves to DFHack's Steam Cloud to enable proper Steam Cloud support for DF outside of Cloud Gaming. People would love it.

You did what you can. To those poor souls that lose their DF save game by playing it in the cloud. They do get a warning. I tested that GFN has a popup that roughly says "Are you sure you want to play this game as it doesn't support Steam Cloud and you may lose your saves?"

EDIT: Also. My save game has been on my nearby GFN server for over 3 months now. They have something similar to Steam Cloud but it's only stored on that one server and they say it's unreliable as years ago it really was. Saves would get wiped for no reason. Also going to any other GFN server would not grab my save from my local server. I still don't trust it or recommend anyone play DF in the Cloud until DFHack is allowed a pass or nVidia stops suggesting Steam Cloud as a requirement.

EDIT2: Perhaps it's best to not compress the files at all as that often results in a very fast compression/decompression time.

myk002 commented 9 months ago

regarding compressing files - yeah, the requirement would only be that the number of files is reduced. tar cf would be sufficient, if that's supported by the vm

myk002 commented 3 months ago

the zip program has the --filesync option, which will ensure a zip file is up to date with the original uncompressed files. For directories that have not changed or in which only a few files have changed, it is much faster. I'm not sure if we have access to that functionality from the libzip library we already link to, though

myk002 commented 3 months ago

some design thoughts (in progress)

architecture

data

Bulk data is a separate directory from dfhack-config since the usage profile is so different. Separating the top-level directories also gives us the flexibility to independently change the cloud sync parameters (e.g. whether to use Steam Auto-Cloud). Finally, it makes it easier for the player if they want to delete the entire directory to save space.

I'm putting .zip in as a placeholder. The archive format can be flexible. Whatever is accessible/reliable on all platforms, produces reasonable compression to reduce upload/download times, and does not take too long to compress.

The limits we need to work within are:

Each vanilla savegame consists of:

Files for each savegame will be collected in a subdirectory named after that savegame. We will need to split the world files (which can be very large) into smaller chunks to stay within single file size limits. We should choose a small enough chunk size so that it reduces the risk of a failed transfer. All files other than the world file are ridiculously small, but can be large in aggregate. A chunked zip of the entire directory where each chunk is a limited size would likely be sufficient. I was not able to find whether directories count as files for the purposes of enforcing the limit, so let's assume yes for safety.

Therefore, each save will take:

savegames on my drive, many of which were collected from players when investigating bug reports, have world files that range from 260K to 517,000K. The Steam Cloud API has a limit of 100M per file, so let's see where a chunk size of 50M gets us:

on the low end (260K world file):

on the high end (517M per world file)

so we're much more sensitive to file size than file count. we could even lower the chunk size to 5M and be ok (if that's actually beneficial).

however, 18 savegames is not very many. players who run large, older worlds may want more control over which games get backed up. And we certainly need to indicate when limits are being approached/exceeded.

flows

player wants to enable cloud sync

  1. player launches DF
  2. on title screen, player clicks "Enable cloud save"

player wants to disable cloud sync and clear storage

player starts game from fully synced state

player starts game with newer state on cloud

player starts game with newer local state

player saves game from within DF

overlay

auto cloud save vs. managed

This choice has a significant effect on the UX and implementation complexity. With auto cloud save, you specify a directory in the Steamworks app configuration. The steam client enforces that configuration by copying that directory from the cloud to the local drive when the app starts and copying the changed files in the directory from the local drive to the cloud after the app exits.

Documentation here: https://partner.steamgames.com/doc/features/cloud

Allowing the Steam client to handle syncing means that DFHack doesn't have to do anything with network I/O or conflict resolution. This greatly simplifies the implementation.

However, it also means that there are many sacrifices around UX:

conflict resolution

integration with other tools