Supporting executables that can't be launched directly

I'm looking to use Reloaded with a game that can't be launched directly from its EXE. The game in question uses a separate launcher executable and if Steamworks is not active it will launch the launcher via SteamWorks and kill itself. Replacing the launcher is an option, they wrote it in C# and didn't bother to obfuscate so I've already created a modified version that can launch directly to the game in question, but there's still no obvious way for me to hook the game itself to do executable code patching at runtime.

Attaching to an already running process is an option, but this won't be very nice for end users [the way it works now since you have to do it manually]. Pointing Reloaded at the launcher and leaving a thread run to init later might be an option, but then every person who develops a mod for the game in question would need to do the same thing.

Any thoughts on what to do to account for this situation?

Was going to write a stand-alone mod loader for the game, but MainMemory pointed me to this which seems like a better option if I can convince others to use it too.

On an unrelated note, I tried tweeting a question at you about how exactly I should go about doing a find & replace against executable code when said address moves between releases and I just want to search for it. At the moment I'm just enumerating all pages, checking if it's set to Read+Execute, and as soon as I find a page with what I'm looking for bail out, but is there a better way to do this?

Hi there; apologies - I'm a bit sleepy at the moment, but alas, a few things come to mind.

Reacquiring addresses after patches.

Finding memory pages isn't really the right approach to this - searching pages of memory is only useful in a very small handful of situations. There are some very specialized situations where this IS useful such as automatically finding a polnter to an emulator's emulated memory with any emulator version but those cases are rather rare.

In terms of the game, often any changes of the underlying code in general between patches will cause the order of functions in assembly code potentially shuffle - not to mention that merely adding a byte to a single function simply offsets any function following.

The classic tried and tested approach for regaining function addresses (and most misc data addresses) between patches of the game would actually be the process of Signature Scanning; which amounts to looking for a specified pattern of bytes in memory with a set input mask. Specifically the pattern of bytes in memory will be a set of assembly instructions and the input mask would just make the signature scanner ignore any specific addresses contained in the assembly instructions within the pattern.

As many good signature makers already exist; I didn't feel that adding one to Reloaded would be a necessity. What I've personally been using up until now has actually been a plugin for IDA called SigMaker; whose output of mask and byte pattern is in the same format as accepted by Reloaded's Signature Scanner.

What to do.

_You can find Reloaded's signature scanner in a neat tidy class called PatternScan inside the Reloaded.Process.Memory namespace. As for the data/byte dump you want to pass in to the function, the safest bet would be a dump of the whole module. To get the base address and length of the main module, simply use ReloadedProcess.Process.BaseAddress, and for the length you can use ReloadedProcess.Process.ModuleMemorySize._

Now once you know the whole range for the executable there are two approaches, a naive but simple approach and a smart approach with a bit of code.

The naive approach would be to simply mark the whole range as ReadWriteExecute using VirtualProtect in Reloaded.Process.Native as you don't know which pages you're dealing with and copying whole thing to a byte array using ReadMemory, toss into signature scanner and add the base address you got your byte array from to the answer.

The smarter approach would be to filter all of the pages returned from GetPages() with the range you know belongs to your game's executable/module (between base address and base address + length) and then filter out the pages which do not have execute permission. Then you can use ReadMemory to simple read the data into byte arrays and scan for the signature. This way page permissions are fully preserved and you will save time when doing the actual signature scan due to having significantly less to scan through [which in itself is also a positive].

I should probably write a utility class at some point for the lengthier approach but it should be pretty simple. The reason it doesn't exist yet is that I tend to deal with old games without updates; thus this feature is rarely used.

I've also changed the function to use pass by reference rather than value to enhance its performance as I was writing this reply - as only now I've noticed the omission. The next NuGet package version will have it. I know this sounds long to implement but the actual solutions for this are actually very simple - the naive solution around 5 lines of code, the non-naive one isn't very complex either.

If you want to learn a bit more about signature scanning; this old wiki page I would call pretty solid https://wiki.alliedmods.net/Signature_Scanning

Fooling Steam DRM (Your first question)

Generally games which use the Steam API/DRM to check if they are launched through Steam will just restart themselves using Steam, in which case Reloaded would just automatically reattach to the game. This is an extra special case we have on our hands here. That said, I've already planned to look into this at some point as the game relaunching self nullifies Reloaded's unique ability of loading and initializing all mods before the game even runs any code.

This just reaffirms me that it's a better idea to do that sooner rather than later. I have a few ideas and I know a few ways I can fool the DRM already but I'm very tired at this point and I'll look into detailing those tomorrow morning as I'm almost falling asleep; in which case I'll make a part II to this post.

A note about the Steam DRM check The check you are probably having is a very basic Steam API check on whether your local account is currently running said game. This kind of check you can literally fool by replacing the game's executable with literally any application as long as it stays in the background and then launch the real executable.

Fooling the DRM or finding a solution to do that shouldn't be the hard part in itself; just finding the most convenient, least hacky way would be (while retaining the ability of injecting into the game and initializing before it executes anything). I'll probably have something working tomorrow.

Extra Note

I somehow have a strange feeling the game in question we're talking about here is Shenmue; Just a blind stab in the dark.

Replying backwards, but so did you. 🙂 You're right, but it probably wasn't hard to figure out. The source code to the decompiled Shenmue launcher which I've updated to include command line arguments to jump directly into one of the two games is in my commit history. There's no special code in the launcher whatsoever that knows about Steam, so it's feasible to replace.

You've accurately described my dilemma. I could ask reloaded to launch the launcher with my new arguments to directly launch a game, but then it's hooking the wrong executable. Hence why I filed this issue, to see how you'd prefer to see this situation handled. It sounds like you're already trying things, but if you prefer to just dictate how it should work I can take a look and submit a pull request in a week or two. Otherwise if you're going to make changes and don't have the game and want a copy for testing purposes I'm willing to donate one.

Signature scanner is probably exactly what I need for the code patch situation, thanks. It's basically what I wrote, except the hard way because I was searching all pages with Read+Execute for a sequence of bytes. But my approach wasn't working either - I found the signatures I was looking for in executable pages (but not the ones Cheat Engine says correspond to the game EXE...) and managed to overwrite them, but it wasn't working. It's probably due to my flawed approach of scanning every page and using WriteExternal to patch. I'll re-implement it later in the week with PatternScan instead.

I'm not in any rush, just wanted to file this when I was looking at it.

Edit: After posting this, I just learned that a stand-alone mod loader for Shenmue I & II was released less than an hour ago. I was hoping they'd avoid this approach and use a generic one like this since I got the same advice when I was planning to build a game-specific one before release but I didn't move fast enough. Oh well. Even more reason why I'm not in any rush.

Today

Well, it took me a bit longer than I thought to write this; but that's mostly because I winded up tracking down a bug that has snuck in inside a master branch which made Reloaded inoperable outside Visual Studio's build directory (strange, isn't it?). It has also come to my attention that my recently implemented plugin system and GameBanana + Github update system operate in very, very similar matter; I'll be merging those ASAP.

I planned to get on the Steam DRM foolery today; and I will - but it'd only be at the end of the day as I feel that while I'm already working on plugin related stuff; this might aswell take precedence.

Steam DRM Foolery

After a while of thought; I've pretty much concluded that for now - the best logical course of action to tackle this solution would be to simply shim/pretend to be the game executable.

The way it would work is that Reloaded would rename the game's executable and drop a very basic executable with no GUI/Console that launches a process from a shortcut. The shortcut would would launch Reloaded's Loader/Launcher with command line parameters that would instruct it to launch a specific game (sm1.exe / sm2.exe in your case). That executable would then sit idle as a background process in order to fool steam until the launched loader executable from shortcut dies.

This process would be optional and a shim the user could drop into a game directory could be generated with a single button; including the executable rename and insertion of the shim.

This can be simplified as: Reloaded Shim launches Shortcut => Reloaded-Loader => Game

Drawbacks

I can see issues showing up with X64 executables - as Reloaded-Loader would need to restart itself in X64 mode. To counter this, I would need to have the shim poll/look for the executable in the processes list a set amount (e.g. every 5 seconds) and require to find the same game process twice in order to safely assume it has been launched by Reloaded's Loader in X64 mode. This should be trivial but it needs to be pointed out.
It is possible for stupidity of the end user to try launch the game through the shim (Reloaded-Loader pseudo-shortcut), which would launch the game, which would launch another copy of the shim and two copies of Reloaded-Loader would wind up injecting mods into the same game - mods get loaded twice.
In the case of a game like Shenmue that has multiple executables; a custom launcher would still be necessary or the user would need to swap out the shortcut when switching between the two games. I reckon a basic custom launcher could be written in <15 minutes literally.

As for the last point. A universal launcher could probably be made that just lists all Reloaded game profiles, filters the ones in the current directory and subdirectories and asks the user - make that act as the shim. The huge majority of the code is in the individual libraries; also available on NuGet. For the most part, the launcher only handles the GUI logic and the loader is a DLL Injector.

Extra Notes

The loader/launcher shortcut is always available in AppData\Local\ReloadedModLoader\ (or well, at least launcher for now, loader in next update).

That said, I'm also up for other ideas; one of them would be simply dll hijacking, like most other loaders do, and then inject Reloaded - but with that we'd lose the benefit of initializing our own mods before the game runs any code.

In this particular test case, both games are x64 executables and 64-bit Windows was a minimum requirement. Actually, whoever coded their launcher made that x86 only, but when I decompiled it and built the replacement I switched it to Any CPU.

Is the 64-bit thing really an issue though? If you know what executable your shim is launching, you can just check the PE header to see if it's 64-bit. Am I missing something there?

I should probably look into the Steamworks API stuff myself anyway. I haven't tried it, but based on the documentation maybe I can just throw steam_appid.txt into the game directory and the problem fixes itself. (Unless that only works for the developer)

I probably worded that wrong there; getting it to work with X64 is actually completely a non-issue, just a small annoyance there in that I'll have to run the loader in X64 mode directly. I can work around that without any issues, I just didn't really know where to put that small tidbit of information.

And yup, I can get that info from the PE header, simply jump to the offset in the file specified at 0x3C (PE Signature offset) and compare the machine type (in little endian) at offset 0x4 from the result of that.

Interestingly enough for some, the way Reloaded's Loader handles this check at the moment is different - it just launches the process suspended and asks the Windows API nicely if it's a 64bit application; I can do either here really.

I'll have something easy working on the master branch by the end of the day, worry not.

(Or to be completely precise, the machine type is in the IMAGE_FILE_HEADER struct of the PE header)

It seems that it's done.

I have implemented initial support of Support of Steam Shimming as of commit b767855dc39aadcdd5f426cbc1080af61348fa89 and then patched a few minor quirks and finalized the job in 31bc41d28517212cac82f924c193ea008907f0d3 , 53e18d6f25725f059fb2d27e896fc3bda5bf82df.

The solution involves clicking a button in the Manage menu, which copies everything necessary to a directory and opens it in Windows Explorer, with a set of simple instructions to be performed by the user.

Exhibit A Exhibit A2

I specifically don't want to automate the process because I explicitly want the user to know what is going on/being performed.

In addition I prefer to avoid asking users or performing any actions that could require admin permissions, generally Steam Games' directories do have the right permissions to allow writes without admin permissions but I wanted to be on the safe side.

I still have to write the wiki page regarding Steam shims (the button on the right of the cursor in the same menu), but for now, it can be safely used by the end user without a problem.

Under the Hood Functioning

The user is required to rename the original game executable and change the EXE path in Reloaded to the renamed executable. Then all that remains is dropping the new EXE in the place of the old one.

Under the hood, the shim reads all game configurations and filters out the ones that are either in the same folder, subfolders or... for brevity parent folders at any level. If only one config is found, it is automatically launched - else the user is asked in a simple menu in terms of which game to launch.

Exhibit B

When there are multiple found games, it looks like this.

Advantages

Reloaded can now boot Steam games directly without having to reattach as the game restarts itself via Steam. This means that for plenty of Steam DRM games, it regains the ability to inject and initialize all mods before the game starts executing any code.
The shim also acts as a pseudo-launcher, allowing you to spoof a regular game's launcher (as seen above).

Limitations

Performing a Steam game update or Steam file verification will likely wipe/remove the shim in which case the user would have to reapply it - not much I can do about that without ensuring I wouldn't need admin permissions. The user is informed of this in the instructions file.

List of Games Tested

I kept it fairly short, just pick games out with unique characteristics:

Sonic Adventure DX (2011/Steam)  [32bit - SteamDRM]
Tales of Berseria                [64bit - SteamDRM + Denuvo]
Wreckfest                        [32/64bit - SteamDRM]

And err... a borrowed copy of Shenmue.

Edit: I forgot to mention.

If you want to test this, for now, this wouldn't work if you have one of the stable Reloaded versions installed.

An optional commandline parameter was added to the launcher. The shim actually checks if the launcher exists in the default directory AppData/Local/ReloadedModLoader/Reloaded-Launcher.exe and asks you for the location if not found.

You can temporarily rename Reloaded-Launcher.exe in there if you want to test out the shim, otherwise the shim would just boot the launcher.

I'm obviously doing something wrong. I installed the latest version and tried to use the Shim following the user steps, but every time the shim launches it just silently closes. As long as the shim is there, the game won't launch anymore.

Application: SteamLauncher.exe Framework Version: v4.0.30319 Description: The process was terminated due to an unhandled exception. Exception Info: System.ArgumentOutOfRangeException at System.ThrowHelper.ThrowArgumentOutOfRangeException(System.ExceptionArgument, System.ExceptionResource) at Reloaded_Steam_Shim.Program.Main()

Based on the code, it's not finding the configuration profiles? What did I miss? I installed the latest build from the releases tab.

Hmm, can't really say much looking at this simple exception message; unfortunately.

Do you think you'd be able to compile the project and send a full stack trace? It's probably some silly generic simple programming error I've not caught somewhere along the way.

Note that this isn't anything specific (Shenmue) as I've had at least two people test it with the Steam version of the game. More likely something like a game profile not being found.

Edit:

Oh, seems I was stupid or tired here and finding 0 profiles is what probably threw the exception: https://github.com/sewer56lol/Reloaded-Mod-Loader/blob/154c2718155ec8650c81803cc9671ae817abf4fd/Reloaded-Steam-Shim/Program.cs#L48

Well; that really is a simple programming error. I am kind of embarrassed really. I'll fix it soon-ish.

Yeah, but why does it find 0 profiles? What did I forget to do? I launched the game from a profile, which would cause it to launch the launcher, but then the launcher can't find the game that launched it.

That is a good question; the actual code responisible for retrieving the configurations is here.

All it does is basically return a list of all game profile(s) with the following criteria:

The executable location or folder is not default.
It is not the "!Global" profile that applies to all games.

And it must satisfy any of the following

The game directory and the current directory are the same.
The game directory is a substring of current directory (game directory is a parent folder of current dir.)
The current directory is a substring of the game directory (game directory is a subfolder).

All comparisons are done with full paths; therefore relative directories should also match.

*Current directory is the directory of the pseudo-launcher a.k.a. steam-shim.

It does feel odd that a profile would not be picked up and I don't really know what to say - I'm unfortunately not even home until next week.

Going through post history; the setup that one of the Forklift people used for playing around with this for fun looked like this:

where they used the "Shim as alternative executable method" and renamed the old Shenmue.exe as Shemue-Reloaded.exe and placed the shim as Shenmue.exe in place of old renamed file.

When writing and testing the Shim; I used the other method whereby I replaced SteamLauncher.exe instead - my setup for the Shenmue profiles was however different in that I had the relative path as sm1/Shenmue.exe. I set game directory to the parent folder instead; *but I did remember testing with the game directory set to the folder containing Shenmue.exe too.

Let's hope this one doesn't narrow down to being something as silly as hitting "Save" in the "Manage Menu".

Oh and I fixed that little exception, as in - it tells the user no profiles were found the way it was originally intended to be in the first place. The severity of it is rather minor and thus I wouldn't be rolling an update specifically for this change at the moment. I'll do so after I do another thing or two once I return home.

On an extra note: The Global mod list also has a mod that hooks Steam's API now and uses a developer-intended feature to initialize Steam through steam_appid.txt.

It works on a game by game basis - depending on developer implementation and Steam features used. Does it work on Shenmue? I feel like it probably would and that hook is the easiest option honestly.

I tried every possible combination of things (shim-ing the launcher, the game executable, relative directories, etc.) and I couldn't get it to work. So I grabbed the latest code and built my own copy.

The problem is that the configuration check on line 87 is just ==, so it's case sensitive. When I run it, currentDirectory is set to D:\Programs\Steam\steamapps\common\SMLaunch but gameFullDirectory is set to D:\Programs\Steam\SteamApps\common\SMLaunch so the comparison fails.

One option is to change line 87 to if (gameFullDirectory.Equals(currentDirectory, StringComparison.OrdinalIgnoreCase) || gameFullDirectory.IndexOf(currentDirectory,StringComparison.OrdinalIgnoreCase) >= 0 || currentDirectory.IndexOf(gameFullDirectory, StringComparison.OrdinalIgnoreCase) >= 0)

Or just call toLowerCase() somewhere. But that's not really a good solution either, because recent versions of Windows 10 allows you to enable case sensitivity on folders to have 2 paths where the only difference is upper/lowercase letters. I'm not really sure why the case difference exists in the paths. I added the folder via the UI and the actual path is CamelCase like that. Did not trace further back since making it case insensitive fixes my problem and who's going to use two files of different case anyway.

I have the shim working now with the code change above. Not sure if you want to keep this issue open until a non-beta release. I'm still looking into the unrelated matter that I asked about for pattern matching, as it's an odd situation and I might need to take a different approach. My original plan was to perform the same hard-patch that works on the EXE binary in memory, but I'm not finding the same pattern doing a scan on the pages containing main module memory region. I'll dig into that sometime over the next two weeks, I probably need to verify with Cheat Engine that what I'm looking for is actually there.

Submit your snippet which ignores case in string comparison as a PR; I'd feel bad for stealing a fix to a bug for myself :P.

While not a perfect solution - there is a very high probability that simply ignoring case would simply bring in much more good than harm. In addition, the default file redirector mod actually performs case ignoring string comparison.

Should a Win 10 user with two folders of same name and different case find this bothersome - they will probably open a PR regarding this. However as things are; given that the redirector has ignored case since the very beginning and nobody has been bothered by this - I'd rather say that this is a fine solution for now.

As for the next stable release: I plan to do that right about now so probably after the commit that fixes this.

In the last week I didn't get any significant reports* which implies that everything's probably fine.

*Apart from the fact that I botched 2.0.0-beta4's installer with wrong directory for default themes to copy and not shipping the default plugins by accident - I found this earlier today.

Thanks again for your help. The stuff was working for me after that last post, but I've yet to revisit this due to other projects. Hoping to try out some neat ideas in the new year.

Sewer56 / Reloaded-Mod-Loader