Is there a way to use BLT to call native methods?

[polarathene]

TL;DR: I'm looking into calling some native methods via exposed C API. I'd like to provide a library/dll file that I can call into from Lua. Would the BLT hook be a way to go about this?

---

The existing FFI solutions I've glazed over is difficult to tell if I can release binaries that users don't have to compile/install beyond drag/drop to PD2 directory.

Besides those two options, I guess I could also do a local web server to redirect to my native code, but if I can't automate that running when PD2 is it's probably inconvenient for users too.

I'm wanting to reduce maintenance of my BigLobby project by avoiding the pdmod which breaks with most game updates. I've looked into how PD2 is handling network packets which is UDP via SteamWorks, packets are identified by an id incremented by message elements in the network.network_settings XML which I currently patch, so a proxy server that manipulates the packets isn't going to work either.

I did try BLTs network message feature early on in my project but it proved unreliable at delivering messages effectively from memory. Does the way BLT work with it's hook make it possible to call native code from another library easily if that's also provided as a DLL(either through BLT or using the same technique)?

[RomanHargrave]

Well, yes. But you're going to have to write wrappers around your function calls to go between C(++) and Lua using Lua's attrocious (imo) but simple FFI interface.

BLT has no actual mechanism by which to load a new native library at runtime, and I don't see the need for one soon.

Now, as the primary developer of the Linux version, I would like to discourage you from shipping native libs, but you are welcome to look at blt4l for examples because the code is just a little bit more clean than the windows version, simply by nature of not using windows API's.

[polarathene]

@RomanHargrave What are your suggestions about approaching this without shipping a native lib? I'm a Linux user myself so they'd be the .dll for Windows users that is drag/drop and the .so for Linux users that just have to add to the LD path like blt4l does. It would be a library/mod that others can interact with to add their own network methods avoiding the problems that present approach has.

I'll be writing the code in Rust exposing a C API. This is easy enough, not familiar with the FFI interface Lua is providing that you mention, unless OVK have included the luaffi library I've come across. Any information on that would be appreciated.

From what I understand some users relay network messages through static Steam IPs for their p2p API. This probably means to support those users I'll need to provide my own relay server(Last Bullet might be able to help with this).

BLT has no actual mechanism by which to load a new native library at runtime

I don't think I need this as mentioned above, wouldn't the methods be available via FFI? I just need a way to bridge Lua calls to my native lib and back.

[RomanHargrave]

@polarathene I didn't mean to say FFI so much as the stack-centric native call interface. If the code is open, that means that the problem of platform incompatibility is theoretically eliminated.

Moving on, since I'm not terribly familiar with the BigLobby mod, nor its workings, I can't make any specific suggestions about ways to go about transmitting data between users, other than perhaps looking in to what exactly causes reliability issues with the user-to-user messaging interface in BLT.

That being said, one potential option is to expose an interface to load native code and then register Lua native functions corresponding to certain symbols exposed by that code. That being said, such a prospect is rather disturbing, moreso given the nature of game mods on windows being very secretive and potentially malicious (HoxHud). Another option is to root around in Diesel to see if we can expose some lower-level native networking behaviour.

[polarathene]

My code would be open source, as is the majority of code I put out there :) If it's worthwhile, perhaps BLT could incorporate what I'm trying to do as a part of it? Not sure if anyone using it could do anything malicious then?

Network methods get defined in an XML file, that gets parsed and methods generated(perhaps with something like googles protobuf .proto format), UDP connections are made with some network hole punching to avoid issues with NAT and not needing to port forward(what the game currently does through Steams p2p API). On the lua end, you provide a handler class to respond to network messages as they arrive, and would modify the network Send function to redirect custom network messages to this native code instead of going through Steams/PD2's network class.

Existing BLT method uses a hidden channel in the games chat system to send data via strings.

[RomanHargrave]

Perhaps if it were more generally implemented as a P2P RPC system, it would be useful for other mods. Since that's pretty much what you described, generalising it and adding it to BLT's responsibilities would be far more in scope.

[polarathene]

Yes the game refers to it as RPCs, I'm just not sure how viable it is to use the existing RPC for each peer in the game as I believe these are provided by the blackbox Network class outside of Lua. As far as I know, what I'm planning to do is generalized already? I'm wanting to support additonal methods for my own mod, but using the existing stucture/architecture that PD2 uses itself in Lua, just not using the existing Network class which handles additional logic and routes messages through Steams P2P API.

I'm happy to contribute to this if assistance can be provided on the BLT side of things. I'm not very experienced in C/C++.

Looking through your hook, Subhook is supported on Windows as well, any reason there isn't a crossplatform BLT lib?

[RomanHargrave]

@polarathene RE crossplatform BLT Primarily because BLT for windows came first.

Ultimately, much of the logic could probably be merged, but the way that the libraries find the methods to hook is very different. On Linux, the game includes debugging symbols, which means that we can just get method addresses from the linker. On windows, the game does not include such symbols, which means that those addresses must be found via a search thru the executable segment.

---

Ultimately, blt4l uses few C++ features, save for where i blatantly copied things out of blt for windows. The architecture is very uncomplicated, and blt4l is very stable right now, meaning that opening PR's or collaborating to build an RPC system is definitely possible.

Currently, there's no real system in place to insure feature parity between the linux and windows parts. Now, that being said, I would definitely be interested in a Windows-familiar developer if we wanted to look at supporting both platforms and reusing the BLT API implementation between the two. I personally cannot build the windows portion myself because I do not have a windows machine, and I like to avoid Windows - let alone developing windows software - like fire.

[polarathene]

@RomanHargrave I'll look through the repo's and try get familiar with the code. I'm curious how well it'd port over to Rust. I'm planning to rely on Crates(language packages in Rust) to provide the majority of the network features. It can compile native libraries for both platforms, if Windows specific code is required should be able to include that as well. I have a Windows VM that can do windows builds if needed.

Feel free to add me on Steam and discuss: http://steamcommunity.com/profiles/76561198052399869/

[SirWaddles]

A network interface between BLT users isn't out of the question. I'd just be going for a message queue as opposed to anything even remotely resembling RPC, there's just no need for that.

I don't really know enough about PD2 networking though, can we establish that kind of connection at all?

[RomanHargrave]

A message queue would be a good idea, and RPC could easily be made to sit on top of that for compat.

Hole punching is still a question. If we can get info about steamworks networking, that would be useful.

[polarathene]

Some peers need to relay messages through Steams IPs, it might be required to provide that via a webserver(last bullet is happy to help here) if unable to use SteamWorks API. The UDP transport that Steam uses I've read is more TCP like in nature. WireShark also shows rather excessive STUN bind requests/respsonses to do the hole punching.

It would be easiest to maintain if the existing infrastructure could be used. This would require patching the XML asset before it's loaded at runtime by the native Network class Diesel uses. If you implement your own additional network class and can utilize SteamWorks P2P API effectively, that'd work too.

Far as I recall, network calls are all handled via NetworkPeer:Send(), when receiving native Network class receives the packet data, matches the packet id, then unpacks the data to send as params to the network handler class bound to that packet id(unit or connection are defaults, I've supplied my own). I can handle everything on the Lua end.

As the packet id is equivalent to the network messages element order in the XML, players with different network methods from other mods could result in each player having a different packet id assigned for the same network method. Eg, P1 has methods A B and P2 has B C, when P1 sends network message B to P2, it's identified as C on P2's setup.....obviously an issue if you were to patch the XML definitions asset at runtime to use PD2's existing network infrastructure.

[RomanHargrave]

I'm honestly surprised steam doesn't do something like n2n and use a p2p VPN over a hole-punched connection.

[polarathene]

@RomanHargrave It seems similar based on the readme for this implementation: https://github.com/patr0nus/n2n

I'm not sure if that's how it works with Steam or just TCP for other platforms. The network code has a specific port to broadcast servers and discover them from, from there p2p connections are initiated. For many peers it's direct IP connections but some need to relay thorugh Steams servers it seems.

[RomanHargrave]

heh, I didn't realize that another p2p software had that name.

http://www.ntop.org/n2n/ is what I was referring to.

[SirWaddles]

@polarathene We could just add one message type for all BLT messages and add an interface so that people can register listeners on string message identifiers and BLT just passes the message along.

[polarathene]

@SirWaddles As long as it's compatible with the existing system that works for me. If you take a look at the current network message definitions, you'll notice that messages vary with things like:

• Delivery reliability(possibly something to do with TCP/UDP protocol based on importance/type of message along with affects on latency),
• A receiver which is basically an event handler, passing the params unpacked to a method with the RPC of who the sender was as the last param.
• Some methods are restricted to client or server sending/receiving only.
• Parameters have types for encoding/validation and some have logic like min/max values.

Majority of all that is handled by Network class on the engine side. I have no idea how the unit type is encoded but it ends up as a two byte value. I've not been able to make sense how that value in the packet is created, but I'm assuming it's some sort of lookup like a hash. An alternative might be to use the units id() method which provides a number that I think is between 0 and 4096, provided the receiving peer has that unit, you can loop through all units looking for a matching id value and continue from there.

If the packets you deliver with this are all string based that would probably impose some performance penalties(encode/decode + latency)? I'm not sure how that differs much from the current private chat channel method.

---

It'd be better to just patch the XML asset before it's accessed with generic methods in that case. If you really wanted to get creative with it, a simple API exposed on top for the developer to use addUnit(tazer_unit:id()), addString("It's a tazer"), etc) that kept track of params used could probably match the packet to be sent to the correct generic method using something like a bitmask.

Alternatively you do some post processing prior with developers messages defined in another XML file like the original network methods parse that and map to generic methods where possible so that in their mod they just call something like BLT.Network.send("unit_and_string", tazer_unit:id(), "It's a tazer").

[SirWaddles]

I was more intending to leave most of it up to the developers, and I don't really expect people to lay out their network functions inside of aggregated XML files.

I think we just need one extra network message in the XML, with ordered delivery (If someone really needs to save the latency caused by dropped packets over TCP, they can use their own networking functions, it's basically a non-issue) with two parameters, a string identifer and string contents. The encoding is up to the developer. The identifier is just a shorthand name for the message type "ChatMessage" for example.

In your mod init scripts, you do something like

network.register("ChatMessage", function(message, length)
    local fromId = message.ReadInt16()
    local text = message.ReadString()
    local messageObj = network.CreateMessage("ChatMessage")
    messageObj.WriteInt16(foobar)
    messageObj.WriteString("This is a message")
    network.SendToServer(message)
end)

[polarathene]

@SirWaddles I understand the XML definitions probably don't benefit any other mod devs if you're not utilizing the XML file that PD2 uses for it's network infrastructure. As my mod heavily relies on tweaking a few of the param constraints, I'm not sure how affected they would be if they weren't consistent with specific settings for the packet or params. Lag can already be bad enough that any high frequency packets being encoded/decoded as strings may cause more latency. My only options presently are patching that XML file or utilizing BLT for some other network implementation, which probably can't handle units as parameters(no idea if my proposed solution is going to be bug free).

My own needs aside, that example looks good. You should be able to infer the local peer id rather than require though(assuming that's the what the WriteInt16(foorbar) is. What makes it any different to the current approach BLT uses with the hidden chat channel sending strings? Looks similar to how I remember when I sent JSON strings in my first version of my mod.

[SirWaddles]

I can't imagine there'd be much lag from writing different types into and from a string. There's no parsing going on, no structure to preprocess.

I don't really know much about the hidden chat channel method, but given that it's for chat it seems like there would be some overhead with things already hooked into it. Using the underlying network functions just removes some of that, but this is largely speculative. Is there any reason you're using the XML definitions as opposed to the chat channel?

[polarathene]

@SirWaddles

Is there any reason you're using the XML definitions as opposed to the chat channel?

Yes, I initially tried JSON strings through the chat channel for sending params(strings/numbers). Sometimes I'd have delays of over 30 seconds to 5 minutes from memory, can't recall but some messages might not have delivered(I remember peers above 4 could not see chat from others but me, but that might have been another connection issue caused elsewhere).

I also need to use the XML definitions because I'm providing modified original network messages(increasing the allowed max peer value). For proper compatibility with what the handlers expect along with logic like min/max, it's much easier to use what is already defined in the XML, makes diffing changes easy too. It also meant no work around for the unit type which I have no idea how that value is figured out(happens in the engines network class).

I wouldn't know what lag to expect(Assuming it wouldn't have the same problems as the hidden chat channel method), what I do know is that strings will make for larger packets, long numbers for example(steams 64-bit user ids) are much larger as a string vs 8 bytes. Quite a few number params would be 2x the size. I've not looked over which packets are sent frequently, but ones like movement for example which are frequent and send vectors(x,y,z) are a good example of packet size being much larger and frequent likely resulting in lag.

[SirWaddles]

That's not what I mean. An 8-byte integer will still be 8 bytes. I'm just using the string type in the XML to allow a variable amount of data, think of the string as just a binary byte array. For example, the ReadInt16 function will pull the first two bytes from the string, convert it into the integer (which is a recast, no parsing required) and then advance the read 'head' 2 bytes.

and honestly, even if it were parsing ints as actual strings, the time taken to do that is in the microseconds. There would be other reasons for delays of 30 seconds.

[polarathene]

Ok so this steam id for example: 76561198052399869 (17 digits). Would be using the same bytes in the string: 0x1100001057DDAFD without casting resulting in a gibberish looking string?

What about when combined with an actual string? String length in bytes + 1 or more bytes appended to indicate read length?(This is how strings are handled in the current packet data). So the only overhead that should actually exist compared to other packets would be an extra byte or two in the packet data for the length of this entire string being sent(a 300 length string for example would have first two bytes in the packet to indicate string length(our string + 2 more bytes for length = 302), then the string value will start with two bytes describing the length of string we actually sent).

Here are packet definitions that only have a string param, you'll notice that several could be used to get the different delivery types or client/server flow restrictions when required. You'd just need to modify the connection/unit receiver handlers to detect the string is intended for BLT use and not the original network message that was hijacked.

<message name="request_player_name_reply" delivery="ordered" receiver="connection">
            <param type="string" />
</message>

<message name="set_trade_spawn" delivery="ordered" receiver="unit" check="server_to_client">
            <param type="string" /> <!-- Criminal name -->
</message>

<message name="lobby_sync_update_difficulty" delivery="ordered" receiver="connection" check="server_to_client">
            <param type="string" />
</message>

<message name="sync_award_achievement" delivery="ordered" receiver="connection" check="server_to_client">
            <param type="string" /> <!-- Achievement ID -->
</message>

<message name="client_used_weapon" delivery="reliable" receiver="connection" check="client_to_server">
            <param type="string" /> <!-- weapon id -->
</message>

<message name="sync_used_weapon" delivery="reliable" receiver="connection" check="server_to_client">
            <param type="string" /> <!-- weapon id -->
</message>

<message name="killzone_set_unit" delivery="ordered" receiver="unit" check="server_to_client">
            <param type="string" />
</message>

<message name="sync_show_hint" delivery="unreliable" receiver="unit">
            <param type="string" />
</message>

<message name="server_unlock_asset" delivery="ordered" receiver="unit" check="client_to_server">
            <param type="string" /> <!-- Asset id -->
</message>

<message name="statistics_tied" delivery="ordered" receiver="unit" check="server_to_client">
            <param type="string" /> <!-- Tweakdata table name of tied down unit -->
</message>

<message name="bain_comment" delivery="ordered" receiver="unit">
            <param type="string" />
</message>

Strings might be limited to a length of 255 characters/bytes, for the few messages that require more than that such as the following packet description, you might need to use the type longstring:

<message name="set_unit" delivery="ordered" receiver="unit" check="server_to_client">
            <param type="unit" />
            <param type="string" />                <!-- character name -->
            <param type="longstring" />            <!-- outfit_string -->
            <param type="int" min="0" max="100" /> <!-- outfit_version -->
            <param type="int" min="0" max="4" />   <!-- Peer id. 0 means its an AI -->
            <param type="string" />                <!-- team_id string -->
</message>

<message name="join_request_reply" delivery="ordered" receiver="connection">
            <param type="int" min="0" max="9" />   <!-- Reply info 0 == may not join, 1 == ok, 2 == kicked, 3 == game started, 4 = map not owned, 5 = full, 6 = low lvl, 7 = wrong game version, 8 = authentication failed, 9 = banned -->
            <param type="int" min="0" max="4" />   <!-- Peer id ~=0 when accepted -->
            <param type="string" />                <!-- My character -->
            <param type="int" min="1" max="127" /> <!-- level index -->
            <param type="int" min="1" max="8" />   <!-- difficulty index -->
            <param type="int" min="0" max="4" />   <!-- Game State ~=0 when accepted, lobby, in game etc -->
            <param type="string" />                <!-- Server character (it was Mask set) -->
            <param type="string" />                <!-- Steam User ID (used for pc only) -->
            <param type="string" />                <!-- Mission -->
            <param type="int" min="0" max="127" /> <!-- Job id index -->
            <param type="int" min="0" max="12" />  <!-- Job stage -->
            <param type="int" min="0" max="4" />   <!-- alternative stage -->
            <param type="int" min="0" max="127" /> <!-- interupt stage level index -->
            <param type="xuid" />
            <param type="longstring" />            <!-- Authentic Steam Ticket ID -->
</message>