Decoupling stash and saving from direct server connections (future)

[12foo]

Hey,

it's been a while. Sorry for my sudden silence on the other issues; I got pulled away by some other stuff. I'm not sure I'll have enough free time to help on the big Season 6 rewrite, but I had an idea about it that I just wanted to write up somewhere. Basically, a lot of the issues stem from the fact that game data syncing is hacked into the game client at a deep level and is fairly complex. It's compounded by being synchronous with the server, i.e. server breaks for a second, game also breaks. So how about decoupling the local syncing of the game state (stash/character) from server side syncing? Very roughly, it could work like this:

• Game client interaction with the server is limited to chat, and specifically when starting a game session (loading a save) or quitting game. Server communication is removed from all other hooks (except maybe trading related stuff).
• At the start of a game session, the client obtains 2 public keys from the server for signing and verification respectively.
• Whenever the client saves character or the stash, it always saves a timestamped set of BOTH character and stash data, signed with the signing key.
• A separate process running outside the client takes care of syncing with the server. It uploads only the signatures and stash changes.
• Whenever receiving new signatures, the server countersigns with the private verification key and the syncing process downloads the countersignatures and applies any stash changes (exact algorithm TBD).
• On loading a save, the client refuses to load the file unless a correct countersignature is present. This ensures that the files on-disk can be trusted (unless someone is proficient enough to modify the keys in game client memory, a tradeoff we accept).

This could also be extended in several ways, like offering save/backup slots for full character data on the server, or just being able to turn Steam Cloud saves back on (I think many people would like that). As long as we perform any processing on-disk and outside the loop, the processing can be arbitrarily complex.

[Dingf]

No worries, I've also been busy and haven't had much time for development as of late. The main issue with decoupling the local sync from the server sync is that the sync happens at the exact moment it's required (e.g. opening the stash or loading a character), and the synced result is then required for the client to proceed normally. This is mostly to prevent timing issues wherein the user could get a synced result from the server, the client trusts the result, and then the user overwrites the trusted result with one of their own which is then synced to the server.

As such, even if the local sync was detached from the server, there would still be a delay on the client side if the server was slow/experiencing latency. For example, the upload/download process would need to wait on the server to update the local files, and then the client would need to wait for the local files to finish updating before proceeding with the load.

Unfortunately, I don't see a good way around the timing issue currently. The alternative would be to set up some sort of polling or file monitoring system for synced files and then download new files if the signatures/checksums don't line up. This however, is both resource intensive and also not as secure since there is a break in the flow of execution between the sync and the loading of the synced file.

[12foo]

I agree, and that would be the point of using cryptographic keys here. In this model, trusting the server at every instant is not required (the keys replace server trust). There is no server network code in the client whatsoever, aside from starting a new session and retrieving the keys. The client can simply operate on local files, and the sync process will occasionally sync with the server. It is entirely decoupled, as if the player is playing locally, without an online connection.

The tradeoffs necessary for this are:

• The client can only load game states that have been verified by the server.
• Any changes to stash or characters invalidate the local state, meaning if the game crashes before the current state could be synced, the changes since the last verified sync would be lost.
• Stash syncing for trade is not instant (every minute or so maybe), but could be specifically triggered by a client command.

Basically, a timeline of actions could go like this:

Game Client	Sync Process	Server	Valid?
Load Game	✓
Take item from stash
Run a dungeon
Put Items in Stash
Idle	Sync Signatures	Sign Signatures
Idle	Sync Countersignatures	✓
Idle	Deliver trade item	✓
Idle	Sync Stash & Signatures	Sign Signatures
Idle	Sync Countersignatures	✓
Take item from stash
Quit game (force sync)	Sync Signatures	Sign Signatures
	Sync Countersignatures	✓

In this model, only the game states marked with a ✓ could be loaded again-- if the client tries to load a save state unverified by the server, it would fail. Since files on disk are cryprographically signed, they can also not be modified by the user (the client will refuse to load them as well). The assumption is simply that the files on disk are always correct and trusted, and syncing with the server is only necessary every few minutes, or on quitting the game.

The benefits of this would be essentially:

• Game operates at local speed on local files, so zero lag (apart from fractions of a % of CPU for the local encryption).
• Network (except chat) is entirely decoupled from the client, leading to fewer crashes and instability.
• Usually just a signature (256 bytes long) is synced, vastly reducing server network load.
• Since everything is stored on disk again, Steam Cloud or OneDrive can be used without impacting the game.

[Dingf]

In this proposed solution, if we are periodically syncing with the server (say every few minutes or so), then shouldn't subsequent actions before the periodic sync cause the game state to be unverified and fail to load? For example, if someone opens the stash, withdraws an item, and then closes the stash and tries to withdraw again a few seconds afterwards. If I'm understanding the model correctly, the second attempt should then fail because withdrawing an item invalidates the game state which prevents the client from loading the stash data. And if so, would the client then be effectively "locked out" of that resource until the periodic sync happens?

I'll probably need some more time to think about what's the best way to implement server sync for the next season, as I haven't really had the chance to do much research on other options yet.

[12foo]

Oh, I should have explained that better. I'll go into a bit more detail as well. When I say "loads the file", I really just mean at the time the player loads the game from the main menu. We basically have 3 states of the game files:

• completely valid (signature and countersignature present)
• locally valid (signature present, but has not been synced yet-- no countersignature)
• invalid (signature missing or does not match files, files have been modified by hand)

The signature is always written to disk at the same time the stash and character data is written. If the stash changes, the game can simply load the stash from file and verify with the local signing key before re-writing the file to disk. The player could interact with the stash several times without contacting the server, but every time, the local stash gets verified and signed. This creates a locally valid save state.

Every few minutes or on a direct trigger (quitting game, etc.), the most current signature is sent to the server, along with any stash uploads/downloads, season achievements, etc. The server countersigns these after processing stash and achievements. This creates a completely valid save state.

On loading a game from the main menu (starting a session), only completely valid game states are accepted, i.e. states that the server has processed and verified. Starting a session is also when the game client receives the keys for that session from the server. The keys are never written to disk, only kept in memory. So as long as a session is running, it's fine for the client to update its local stash with the signing key without contacting the server. We use this signing key to ensure that game data on disk doesn't differ from game data in memory while the game is running. The server countersignature is so the game data on disk cannot be changed between sessions without the server having seen it.

There is an attack vector here: if someone has the skills to intercept or replace the keys on the wire or in game memory, they can cheat the server as they like. That's about the same skill level it would take to cheat the game in its current state as well, though.

I really should try and see if I can get a proof of concept going for this, but probably not this or next month. :(

[Dingf]

I see, that makes a bit more sense. I'll try to see if I can get a rough implementation working when I have some time available, though likewise I also probably won't be able to get around to it for a while.