Linux Support

[karasuhebi]

Is this planned? :D

[BangL]

as far as i know it wouldnt be that easy.. for now we are "hacking" the "IP Helper API", which seems to be windows-only. so on linux, we would need a completely different concept. a simple "port" isnt possible. so don't put your hopes too high. ;)

[karasuhebi]

Oh sorry I didn't mean to imply it'd be easy, I was pretty certain it would not be something trivial and would definitely require some work. Thanks for confirming my suspicions though. Here's to hoping it happens though! :)

[BangL]

well.. don't get me wrong. i am totally hoping the same! too bad i have no idea about C/C++, otherwise i would try to help

[polarathene]

So something like this?

Could you list what would be involved/required for someone to tackle a linux hook. It looks like you are mapping the lua methods like briefly covered here for making a hook, what else would the linux version need?

Besides the file stuff, how much of BLT's codebase would you say is shareable(platform independent)?

[ZNixian]

Wow. That actually looks fairly simple - just 'override' the calls used to the the Lua VM, and inject something to load out of user-defined files.

[SirWaddles]

I haven't looked at Payday's linux version just yet, but if it's anything like the Windows one, I don't think that the LD_PRELOAD environment variable will be adequate.

Given that Payday has its own version of the lua lib, I doubt it would be being used a shared library. So in the final compiled version, calls aren't being made via the function symbol, they're just going straight to the function address in the binary, so compiling a shared library with matching symbols I don't think will be able to intercept anything.

and if Overkill have stripped the binary (not unbelievable given it's their proprietary info), it makes this a lot more difficult.

In all the examples I've found regarding the LD_PRELOAD trick, including the one that polarathene mentioned, only system calls are intercepted, things that are strictly shared libraries.

I hope I'm wrong, I've not looked into it extensively. I'd like to support linux, but I'm not really sure how.

[polarathene]

I have rather limited experience/knowledge in this area. I've had luck manipulating games with CheatEngine on Windows in the past, a common approach is to locate a static pointer address in memory to override a value or function(alternatively hook it).

If unable to do that because the target data is dynamic due to being created on the fly from something like Lua, you could try your luck with Array of Byte(AOB) scans. The latter is a method I used for Payday 2 before being aware of Lua hooks.

I imagine it's much harder to locate the actual Lua functions addresses in memory, and that they're likely to change with updates(AOB scans might help here). If the addresses in memory were found, would the trick I linked be viable, or the addresses useful for mapping/hooking in some other way?

If this approach could work, how does compiling lua into an application affect it when loaded into memory, is there a chance of identifying/matching similarities of the version of lua used to another program that embeds lua into it's compiled application? Or would the binary data related to lua be completely different between the two compiled applications? If the former we might have a better chance of isolating function calls to match with PD2's lua in memory.

[ZNixian]

ldd on payday2_release (the linux executable) no liblua.so.1 in sight...

I wonder if we can do something with these, particularly glibc.

linux-vdso.so.1 libGL.so.1 libopenal.so.1 libsteam_api.so libm.so.6 libdl.so.2 libgcc_s.so.1 libpthread.so.0 libc.so.6 libnvidia-tls.so.352.63 libnvidia-glcore.so.352.63 libX11.so.6 libXext.so.6 librt.so.1 libstdc++.so.6 /lib64/ld-linux-x86-64.so.2 libxcb.so.1 libXau.so.6 libXdmcp.so.6

[SirWaddles]

Hey, while you're here, can you run the 'nm' utility on the binary? It should show what symbols have been exported. Probably do a pastebin dump or something, because there's likely to be a few.

See here

[SirWaddles]

@polarathene We're already using signature scanning on the windows one. I highly doubt that the signatures will exactly match between platforms, so we'd have to find new sigs. As well as we'd need ways to find the process handle (using a windows.h function at the moment) and to replace some existing functions (using Microsoft Detours at the moment)

Even then, the real trick is getting into the same process space. At the moment, we're using a proxy DLL which has worked pretty well.

[ZNixian]

Ok, done the nm dump. If you want to lag your browser, go to http://sprunge.us/JSiI to view the 4MB of result.

[SirWaddles]

@minermoder27 Hey, it's not a stripped binary. That solves a few problems.

[ZNixian]

Sorry, I didn't see your link. Run nm -an | c++filt

Done with those flags at http://sprunge.us/TQSU

[SirWaddles]

That just unmangles the c++ stuff, it's basically the same either way.

[ZNixian]

If only Lua was under the GPL... They would have to be able to give us object files, for our linking convenience.

[polarathene]

So you're doing something quite similar to how CheatEngine manipulates an applications memory? Just you're doing it via things like IPHLPAPI? I'm no help here so I'll leave you guys to it, best of luck :)

If it helps any, when I was googling to see how others were doing lua hooks for games, I think someone mentioned taking advantage of the steam api that the game called externally. No idea if that's the STEAM:some_func() calls I see in lua, might help you pinpoint a lua function though? shrugs

[ZNixian]

I've just found https://github.com/gaffe23/linux-inject We can't use this when running in a different process, but we should be able to run it in glibc, after overriding it with LD_PRELOAD, so we can run it in the context of PAYDAY2. After that, we could call PAYDAY2's Lua functions.

[SirWaddles]

It looks like that could work, but I don't know about having to get players to disable important memory protection features just to get the thing to work.

[ZNixian]

Solution: we don't.

"On many Linux distributions, the kernel is configured by default to prevent any process from calling ptrace() on another process that it did not create (e.g. via fork())." - README file

Therefore, run this from within glibc, from within the process itself. As a process can modify itself, we can leave this on.

[SirWaddles]

I'm kind of confused.

Isn't the point of that library so you can get inside another process space? If you're already there, why do you need it?

[ZNixian]

Actually, valid point. I think I was over-thinking it quite a lot :P

[SirWaddles]

I guess if you hook in via LD_PRELOAD (don't know how user-friendly that process is), you don't even need to proxy anything. Since there's no point wrapping any of the shared functions (they're useless to us) the only problem left to solve is being in the space, and we're there anyway.

We know the addresses of every function, thanks to the symbol exports, so all that's left is we need to find a way to neatly start overriding, and calling functions based on those symbol addresses.

[ZNixian]

ptrace()

see man ptrace or http://www.linuxjournal.com/article/6210 as an example of injecting code into a running program. Pretty much exactly what we want.

[BangL]

i dont understand any word, but it looks kinda promising. if you need a tester, let me know^^

[ZNixian]

TL;DR of the conversation: We've found a way (that should work) to get our code into PAYDAY2. I'm currently writing a test program, though a lot more work is probably needed to actually get things to the point where you can drag in a mod and have it work.

[ghost]

Found this https://github.com/RomanHargrave/blt4l

[ZNixian]

Okay, found https://github.com/Zeex/subhook Very simple way to override methods.

typedef void (*foo_func)(int x);
subhook_t foo_hook;

void my_foo(int x) {
  printf("foo(%d) called\n", x);

  /* Call foo() via trampoline. */
  ((foo_func)subhook_get_trampoline(foo_hook))(x);
}

int main() {
  /* Create a hook that will redirect all foo() calls to to my_foo(). */
  foo_hook = subhook_new((void *)foo, (void *)my_foo);

  /* Install it. */
  subhook_install(foo_hook);

  foo(123);

  /* Remove the hook and free memory when you're done. */
  subhook_remove(foo_hook);
  subhook_free(foo_hook);
}

[ZNixian]

@SirWaddles Got Lua injection working with the Lua command, so I can run arbitrary code within the context of the Lua command, and also presumably PAYDAY 2. As such, all that is left is to port the non-injection parts of BLT to GNU/Linux, which should be comparatively straightforwards.

[SirWaddles]

Yea, most of that is cross platform

[karolherbst]

I could also help to work on this, because I am a Linux dev and own a copy of payday2 :)

[SirWaddles]

Up to you. There's already that port that @goontest mentioned, and I was under the impression that @ZNixian was working on a PR? Not sure.

Almost seems like this one is out of my hands.

[karolherbst]

well I don't care who does it in the end, because I have usually enough to do with nouveau stuff. But I will test it in the end and most likely try to change stuff I don't like

[ZNixian]

What the problem is at the moment (not helped by me never having used x86 assembly before. If you do know assembly, then it would be amazing if you could help. The problem is, when I am trying to inject into PD2 from a shared library (NOT using PIC, by the way - it would break everything). However, the SubHook library I am using only uses a relative jmp command - so it ONLY works if PD2 is loaded within 2GB of the Lua functions we need.

If you could fork subhook, and make it at least kinda work with 64bit absolute jumps, that would make BLT for Linux possible.

[karolherbst]

You don't want to use subhook, because this opens a world of pain.

In the end you want to just intercept the right call to an external library, everything else you can't do reliable.

Now in fact you want to use maybe LD_PRELOAD and just overwrite the thing payday2 calls to initialize LUA, dlopen the real thing (can also be the currently running binary with dlopen(NULL)) and dlsym the real function.

This is the less painfull thing to overwrite external functions.

[ZNixian]

Unfortunately, a copy of Lua is statically linked to PD2. The reason why I was using subhook was I don't know assembly, and it looked like a easy way to do things. If there is a better way, by all means please give me an example I can use.

[Ozymandias117]

I finally found some time to try to help with the hooking situation. I've made a temporary fork of subhook which pushes the address onto the stack and returns rather than uses a jump: https://github.com/Ozymandias117/subhook

See if this works any better for you. Note that I would advise against using the trampoline, and would instead uninstall the hook prior to calling the function and reinstalling it when you're done. This will be slightly less efficient, but should be safer due to the large memory space.

[ljrk0]

Hi all, x-posting from here https://github.com/RomanHargrave/blt4l/issues/3#issuecomment-205339411 and here https://github.com/RomanHargrave/blt4l/issues/5:

I've made it to override "dsl::EventManager::update()", but sadly when calling the original function from within the hooked one it hangs while trying to std::__1::recursive_mutex::lock() -- so we have a multi threading problem. I've no idea about multi threading though so help would be appreciated if you have ideas on how to solved this.

Note: I use @Ozymandias117 subhook version since the other one completely segfaults and doesn't even hook for some reason and you need to un-comment the lines for Installing the subhook in src/hook.cc if you want to 'activate' the feature.

Just compile as described in the README and run:

export SteamAppId=218620
export LD_PRELOAD="/path/to/libblt_loader.so $LD_PRELOAD"
"/path/to/steamapps/common/PAYDAY 2/payday2_release"

Note that because of the way LD_PRELOAD works, no spaces are allowed in the hook-libs path.

For debugging with gdb you can automate this too (call with gdb -x gdbinit.txt)

set environment SteamAppId=218620
set environment LD_PRELOAD="/path/to/libblt_loader.so $LD_PRELOAD"
file "/path/to/steamapps/common/PAYDAY 2/payday2_release"

[RomanHargrave]

To follow up what Leonard said, and from what I have seen of BLT4Win (gah! WINAPI! kill it with fire), I don't know that we need to be terribly caught up with multi-threading concerns as long as we aren't too intrusive WRT the stack, and because for each thread, there is a different LUA context, and we should logically always get the correct LUA context for a thread as a parameter.

Forgive me if I've skipped over or misstated something. I've just gotten off an eight hour train ride.

[RomanHargrave]

@JamesWilko could you perhaps tell me which function you think do_game_update would be, symbolically? I'm having a little trouble figuring out what it is that you're calling, and I would like for optimal compatibility.

[SirWaddles]

I'm not sure really, but it seems to run every frame and also has a pointer to the lua state in its class owner and seems to take eight bytes of data.

Originally, we were using it to get the lua state, but we moved off of that later on. Now it's just being used to call lua stuff synchronously (to tell Payday that the contents of a HTTP request have been receieved, on its own thread)

You could probably just pause Payday in a debugger at any point and it'll be in the callstack somewhere. I'm guessing it'd be called 'Update' or 'Tick' or something like that.

[RomanHargrave]

Then it's probably Application::Update(). But that, nor do many other functions like it take a parameter, other than probably this, which I assume you are referring to.

[RomanHargrave]

OK.

It's definitely Application::update(this).

I was just stupid, and being a C programmer, didn't think about stupid parameter nonsense. I wish C++ still forced you to insert "this" in to the function parameters like the "good" old days.

If it ever did, or I'm thinking of smalltalk or objective-c or something like that.

[Ozymandias117]

I spent a little while last night trying to port this one over. It builds and hooks (mostly) the same functions... The issues currently are: Console was just removed for now (It didn't appear to be too important at the time) lua_newstate is hooked instead of luaL_newstate. The latter did not appear to ever get called on Linux Other than the console output, mods aren't loading, so there are obviously issues remaining.

I created a basic Makefile for now, and once we get it working, there will be effort involved in having both building - there were several places where msvc and gcc disagreed on "valid" code. You can see the current progress on this branch: https://github.com/Ozymandias117/Payday-2-BLT/tree/linux-work if you have any insights.

Build instructions from checkout directory: git submodule update --init mkdir ./build make

It requires the -dev files for libcurl and openssl. (On Debian I've been building with: libcurl4-gnutls-dev, libssl-dev, and presumably build-essential covers anything else)

For running it, I've been using: export SteamAppId=218620 LD_PRELOAD=./libblt_hook.so ./payday2_release

[karolherbst]

sounds good, I will give it a shot tomorrow. Maybe there can be also something be used like systemtap to see what kind of functions are called (for development)? I am sure you can just hook all lua* functions with it, but never used it myself.

[RomanHargrave]

@karolherbst We shouldn't need to hook more than two LUA functions, according to BLT4WIN (this project). Seeing as the LUA library is pretty platform-agnostic, I'm proceeding with the assumption that a lot of this code can just be ported.

@Ozymandias117 I just had a look at your branch, and it looks like you may be encountering some issues due to the way Detours for Windows (TM) works vs what we are doing. DFW actually appears to change the stack, hence the extra bits passed as function parameters.

[karolherbst]

@RomanHargrave yeah I know. I just meant this way a developer could just trace what PD2 is doing on the LUA side

[Ozymandias117]

@RomanHargrave Are we sure about that? Two of the functions that are hooked with Detours, lua_call and lua_close, take the same arguments as the original functions. do_game_update takes some values... but I don't know how those were decided. The only one that seems different from what I expect is luaL_newstate, although I don't know why it's different.

According to the lua docs, it seems like luaL_newstate doesn't take any parameters? However in CREATE_CALLABLE_CLASS_SIGNATURE it's shown as taking (char, char, int), then in lua_newstate_new it's taking (void, int, char, char, int)... That shouldn't even be a class with a this pointer? Both sides also show it returning an int when the documentation says it returns a lua_State... (For reference: http://www.lua.org/manual/5.1/manual.html#luaL_newstate)

@SirWaddles Do you happen to have any input as to why luaL_newstate is being hooked the way it is?

[SirWaddles]

Yea, PD2s lua code is not 1:1 with the lua library. They changed the luaL_newstate function so that they could add their own lua functions and stuff as it gets initalized. We discovered this when we were disassembling the binary.

It's possible that the one we hooked is not actually the luaL_newstate function, but it was where the lua state was being created, and the top half looked a decent amount like it. (Keep in mind, we didn't have any symbols when we were doing this, a lot of it was guesswork. It looked like it was doing the right thing and we hooked it, the result worked so we used it)

[RomanHargrave]

@SirWaddles interesting.

I'll keep that in mind. I imagine that the changes carried to the linux release. WRT _newstate, I intend to hook lua_newstate since luaL_newstate calls it anyways.

[SirWaddles]

Small note, I'm not sure if it's a concern, but we tried that originally and it didn't work for us.

It was because PD2 defines its own pcall and dofile functions outside of lua_newstate, and then later removes them. We needed to grab the state after it did this black magic, because we needed to overwrite those functions in the lua code itself.

I still have some screenshots illustrating this when I was disucssing it with Wilko from when I was working on it back then, but not with me at the moment.