Make a plan for the steam release

[myk002]

Opening this bug to gather ideas for what we can do to minimize the time it will take to support the steam release when it comes out. We of course have incomplete knowledge of exactly what will be required, but it behooves us to prepare as best we can. We should at least have a plan prepared so we can communicate to our users what to expect in terms of timeline and feature set.

Will we support the steam version along with older versions of DF or will we drop support for older versions?

What reverse engineering will we likely need to do before we can make a minimal release? Who will do the reverse engineering and how can we split the work?

Is there any infrastructure or tooling we can prepare now that will reduce the toil later? For example, do we know the toolchain changes? Can we prepare and test the appropriate build environments?

Can we cut out parts of DFHack from an initial release so players can have at least some functionality quickly? What is that minimal set? Can we triage core functionality, plugins, and scripts into "likely to continue working without modification", "may need updating", and "needs rewrite" buckets?

Will we need to change anything about our packaging and distribution? Do we need/want to register as a steam "mod" or "DLC"?

[ab9rf]

It is essentially guaranteed that the Steam release will be built with a different toolchain thanprevious releases. This is because it will have to linked against steamlib, and while Steam does maintain older versions of steamlib for backward compatibility with older games, Stea usually does not allow publishers to publish new games that depend on downlevel versions of steamlib, and I doubt that any of the currently "acceptable" versions for a new release are compatible with VS 2015, which is what Toady has been using. I'm therefore expecting this release to be built with either VS 2019 or VS 2022, rather than the VS 2015 compiler that Toady has been using since 2016. While my preliminary research is that Microsoft has retained ABI compatibility amongst VS releases since VS 2015 (possibly even VS 2013), and so the changes here might not be totally dramatic, it will probably force all of us to update our toolchains (which, to be fair, will be a blessing, given how hard it is to actually get a working build environment today) and will likely complicated reverse engineering because we'll be dealing with a different toolchain. But (unless Toady tells us in advance, as he did when he switched to VS 2015) we won't know what the new toolchain is until we can get our hands on the product for forensic examination.

It's also almost certain that Toady will replace the bespoke SDL1 he's using (Toady had to hack it, slightly, to build in 64 bit) with something else, most likely SDL2. A big selling point of the Steam release is an improved visual experience, for which he is relying at least in part on other developers, and finding people willing (or even able) to develop for SDL1, which is now eight years past EOL, is likely to be difficult. IIRC, SDL1 is also not supported on Steam Deck, and I'm sure that Steam Deck compatibility is a goal that Kitfox has. I also don't know if current releases of steamlib will work with SDL1. All this leads me to anticipate that SDL1 is likely to go away. Since we currently use SDL1 as our code injection hook on Windows, we will probably have to come up with a new strategy for injecting our code into the executable for the Steam release. We will probably, again, have to wait until we have access to the actual release executables (or at least a preview version of them) before we can identify if this has happened, and if so what alternative strategy we can use.

I don't know a damn thing about Steam's TOS, but I do note that I own and play several games that I bought through Steam for which I use third party modloaders that are not in any way registered with or authorized by Steam, so I don't think we'll have to do that unless we specifically want to.

[myk002]

Thank you for that writeup. So what I'm hearing is:

• likely change to vs2019 toolchain
• likely change to SDL2

And the impact of both changes is largely unknown.

I seem to recall that we thought moving to SDL2 would be a good thing for DFHack, though I don't remember specifically why.

So what can we do in advance? Is the plan just to wait until we get our hands on a binary and then react?

From what I hear, the initial release will be windows only. What can we non-windows-using devs do to help?

[ab9rf]

The best thing we can do in advance is try to lobby Toady for information. The "moon shot" is to convince him to give us PDBs, but I doubt he can be convinced to do that. More generally, the more information he shares with us, the better.

[quietust]

I'm 99.9% confident that Toady will not give us PDBs, since those would give us the name of every single function in the whole game (including the names of every class with non-virtual methods). Optimistically, I think the best we can hope for would be a free Steam key or two.

[lethosor]

The only DFHack-specific benefit of SDL2 I've thought of (other than the general improved support for newer systems) is that we could potentially hack into the SDL2 dynamic API to hook in in a more consistent fashion across platforms. Windows still might require replacing SDL.dll, though, since making use of the dynamic API requires setting an environment variable before DF starts.

[clinodev]

With a reminder that I am not Kitfox or Bay 12, Kitfox asked me to moderate the Steam forum, and Valve thinks I'm a Kitfox developer. I have a June 2021 copy of an unreleased internal variant of "0.44.12" in my Steam library which is definitely not the release version of 0.44.12, (new music, old graphics) and which installs and runs just fine through Steam like a normal Steam game.

I know it's Kitfox and Tarn's intention to be helpful to the utility and modding community. Is there anything I could look at to be helpful here?

I see that SDL.dll in the internal release is 300k, the same size as SDL.dll in a fresh copy of vanilla 0.47.05, so very likely the same version at that time, at least. With the exception of files created on first run like "stderr.txt" the filenames in the main folder of the currently uploaded to Steam version appear to be identical to those of a fresh copy of vanilla 0.47.05, which would suggest to me the toolchain is the same for them.

My advice is to just ask Tarn about this stuff. This all looks like bog standard FotF stuff to me.

Okay, I went ahead and asked Tarn:

"So far SDL is unchanged from last release. And it’s still msvc community 2015 too."

Attached a text file main folder file list for the internal version presently on Steam. Steam DF files.txt

[ab9rf]

If the toolchain isn't changing, then our process for engineering the framework to the new release will be essentially the same as for past releases. How long it takes us to adapt will mainly depend on how much Toady changes. Based on his FotF statements so far, I expect substantial changes in UI-related areas, but relatively minimal changes in the game simulation. Yes, there are changes, and some of them (like the labor system) may prove to be extensive, but overall I anticipate that the data structures will not change so much that our prior practices should be adequate to produce a working solution in a reasonable time.

If you can run certutil -hashfile sdl.dll md5 on that sdl.dll file and verify that its MD5 hash is 1ae242c4b94cb03756a1288122a66faf, that would be helpful.

[clinodev]

If you can run certutil -hashfile sdl.dll md5 on that sdl.dll file and verify that its MD5 hash is 1ae242c4b94cb03756a1288122a66faf, that would be helpful.

It is indeed.

[myk002]

Ok, it looks like there will be no dramatic earthquakes just yet. There will be reverse engineering required for the new viewscreens, but it sounds like we won't have to make larger adjustments.