search

SteamRE / SteamKit

SteamKit2 is a .NET library designed to interoperate with Valve's Steam network. It aims to provide a simple, yet extensible, interface to perform various actions on the network.
GNU Lesser General Public License v2.1
2.6k stars 495 forks source link

Possibly integrate SteamDatagram protocol to SteamKit #123

Closed paralin closed 4 years ago

paralin commented 9 years ago

SteamDatagram is a new technology used in the DOTA 2 client, currently only on the US West region.

It allows clients to connect via UDP to proxy servers within the Valve datacenters that will route the messages internally to/from the game server rather than connecting directly to the gameserver itself.

This masks the true IP addresses of the game servers, thus preventing a DDoS attack. I believe the system is capable also of routing across datacenters, which means the DOTA client will select the closest datacenter as well as proxy servers from alternate datacenters, which effectively makes region selection a question of player pool, not networking. Theoretically if the connection between Valve datacenters is fast, you could play on any Datagram-enabled server with the same ping as if it was hosted in the nearest datacenter, regardless of the server's true location.

Although this client is currently built into the DOTA 2 code and thus might not be suitable to implement into the generic SteamKit client, I believe due to the naming convention (SteamDatagram) and potential usefulness of this concept, Valve will likely expand usage of the protocol to other games (CS:Go matchmaking?) in the future. It might be worth investigating integrating it into SteamKit.

azuisleet commented 9 years ago

If SteamDatagram is similar to GC in that it's routed through Steam3, I think it would make sense to support the same amount of functionality as GC.

Netshroud commented 9 years ago

I may be wrong, but I believe that SteamDatagram is essentially a tunnelling protocol for Source game traffic, which falls outside of SteamKit's domain. It doesn't go through Steam3 as far as I am aware.

paralin commented 9 years ago

Yes, it's for Source game traffic, but could be used for anything that uses Steam game sessions... So it might be in the SteamKit domain. Not sure.

Netshroud commented 9 years ago

Closing due to inactivity, but as far as I know this is purely a Source Engine construct and thus outside of SteamKit's domain. If you see Steam itself using this, please let me know.

moofMonkey commented 5 years ago

https://partner.steamgames.com/doc/features/multiplayer/steamdatagramrelay SDR is now available to be integrated not only to Valve games, is there anything new on this topic?

azuisleet commented 5 years ago

Nothing new yet, but if anyone knows which games do use this feature, it would help with research.

moofMonkey commented 5 years ago

At the moment I know only 2 of them - Dota 2 and CS:GO. I've pointed to ability of integrating SDR into other games because @Netshroud wrote "as far as I know this is purely a Source Engine construct".

Some hints about reversing connection:

  1. https://github.com/paralin/Dota2/issues/14
  2. https://partner.steamgames.com/doc/features/multiplayer/steamdatagramrelay
  3. https://partner.steamgames.com/doc/api/steamnetworkingtypes#SteamDatagramRelayAuthTicket
  4. In Dota SDR is handled inside networksystem and steamnetworkingsockets

And some info that I've reversed alongside with @LWSS:

  1. CMsgGCToClientSteamDatagramTicket (k_EMsgGCToClientSteamDatagramTicket) holds CMsgSteamDatagramSignedRelayAuthTicket inside "serialized_ticket" field (and nothing more)
  2. CMsgSteamDatagramSignedRelayAuthTicket holds CMsgSteamDatagramRelayAuthTicket inside "ticket" field (and something else)

Auth tickets logic can be obtained from latest @paralin's Dota2.GameClient version at https://github.com/paralin/Dota2/tree/a55e1a5b6632572129472d800e21c8da806d5838 I think I'll make fork of it soon and upload updated GameClient (now it only lacks connection feature which must be resolved in this issue).

Connection repro:

  1. Open Dota 2
  2. Press "PLAY DOTA"
  3. Custom Lobbies => Create
  4. Specify any server but not local one
  5. Press "START GAME"

What's needed next?

  1. Ping relays, pick one that suits best ping
  2. Connect to picked one and tell it needed info (gameserver id/route/etc?)
  3. Encrypt connection, send/receive messages
azuisleet commented 5 years ago

There are a couple layers to this. SteamDatagramRelay is built on top of SteamNetworkingSockets.

As far as implementing SteamNetworkingSockets, the closest implementation that exists is GameNetworkingSockets. Rewriting a pure implementation would probably take some time, so it probably makes sense to build the missing integration pieces of SNS to GNS.

SNS Steam notes: ClientNetworkingCertRequest, ClientNetworkingCertRequestResponse for the identity generation ClientP2PConnectionInfo for P2P rendezvous. unsure about what is actually contained

Still to do:

azuisleet commented 5 years ago

On the other hand, if this feature is dependent on a C library (GNS), perhaps it would just make sense to use SNS for your project?

moofMonkey commented 5 years ago

GameNetworkingSockets

I think you're right since https://github.com/paralin/Dota2/issues/14#issuecomment-335778828 mentions that packets are encrypted by AES-256, and GameNetworkingSockets URL mentions same (but different type of it - GCM instead of CBC).. and GameNetworkingSockets uses UDP (I've seen UDP packets sent to relay). By the way it solves only point 3 (Encrypt connection, send/receive messages), but not others

  1. Ping relays, pick one that suits best ping
  2. Connect to picked one and tell it needed info (gameserver id/route/etc?)

I haven't seen public code that solves these problems, and digging inside Dota client haven't helped much (maybe SteamDatagramTransport::CSteamDatagramTransportClient::SendPing is used to ping relays, and it uses CMsgSteamDatagramConnectionStatsClientToRouter, which is very likely to be what https://github.com/paralin/Dota2/issues/14#issuecomment-304129467 describes)

moofMonkey commented 5 years ago

Also, GNS have wrapper for C# at https://github.com/nxrighthere/ValveSockets-CSharp

paralin commented 5 years ago

I'm still interested in building a library for SDR. At the time the only blocker was the ticket negotiation as discussed above.

moofMonkey commented 5 years ago

@paralin, only thing I'm afraid of - publicly available full connection to game server may introduce many CS:GO and Dota leveling bots, but there's also bright side of it - many sites/companies will be able to spectate games without wasting human time and make people closer to better custom games hosting (better API, performance) in Dota. BTW I still think that's worth it since we'll be able to RE this protocol faster.

azuisleet commented 5 years ago

None of what is reversed here would make an impact on bots, since all of this functionality is available through steamclient right now. Steamclient has a full SDR implementation and a whole page that documents how to use it.

Anything released from this work would not include game-specific scope. You would be required to implement SDR Auth Ticket negotiation and supply it to the library. The SDR Auth Ticket negotiation is already documented in the steamworks SDK, and comes across from CMsgGCToClientSteamDatagramTicket.

moofMonkey commented 5 years ago

Ok, so I'm using wireshark now. Seems like 0x010x00sdping and some data is ping to relay, and relay response is probably ping to valve datacenters After a ton of pings (10 for me) client sends 4 bytes, CMsgSteamDatagramSignedRelayAuthTicket+some 24 bytes. And relay responses with some 8 bytes and steamid:%server_steamid%+some 11 bytes after that If relay is backup: client sends NETWORK_DISCONNECT_DISCONNECT_BY_USER and relay responses again with some bytes at start and steamid:%server_steamid% If relay is primary: client sends some unencrypted data again and next message (or next to next) is already encrypted

I'll try to debug this better soon by making hooks on UDP receive/sendto.

Also, there's interesting ifdefs in steamworks SDK triggered by STEAMNETWORKINGSOCKETS_ENABLE_SDR (I think most of them are documented btw)

azuisleet commented 5 years ago

As per the steamworks documentation, we know that communication with the SDR is using SteamNetworkingSockets.

azuisleet commented 5 years ago

Alright, so I believe communication between GNS and SNS is possible. You need to attach the cert data to GNS, so here's an example function for that:

https://gist.github.com/azuisleet/e2859ac9f88e1c6dee028e591ed0dc72

Trying to do this from C# seems like a hassle, so we'll have to see what our options are.

The actual process of setting up the connection is simply creating the Ed25519 key and having Steam sign it. Then you pass it along to GNS:

https://gist.github.com/azuisleet/271b712c976e00743896023b37e4664c

From here we can start hooking/dumping decrypted messages since we now know that GNS is suitable for communication with other SNS endpoints (including SDR hopefully?)

paralin commented 5 years ago

I'd be inclined to write this in Go, we have strong support for all the crypto / protobufs and can link with C including SNS.

moofMonkey commented 5 years ago

I'd be inclined to write this in Go, we have strong support for all the crypto / protobufs and can link with C including SNS.

Yeah, me too (at least for better cross-platform compatibility). By the way - what prevents us from porting it to C#, or porting it from Go to C#?

yaakov-h commented 5 years ago

Time and people?

moofMonkey commented 5 years ago

I'd be inclined to write this in Go, we have strong support for all the crypto / protobufs and can link with C including SNS.

Yeah, me too (at least for better cross-platform compatibility). By the way - what prevents us from porting it to C#, or porting it from Go to C#?

Time and people?

Ok, so on what go-steam repo we should rely? https://github.com/faceit/go-steam or https://github.com/Philipp15b/go-steam? Also, should it be new repo or just an issue/PR on go-steam?

paralin commented 5 years ago

Philipp15's is upstream of faceit's.

azuisleet commented 5 years ago

I don't believe GNS will work out-of-the-box with SDR, as SendEncryptedDataChunk needs to attach its own SDR header.

This may be as far as we can support this, since regardless of what language it is written in, the implementation needs to vary slightly for SDR connections.

moofMonkey commented 5 years ago

So, client sends CMsgSteamDatagramConnectRequest after receiving CMsgSteamDatagramGameserverSessionEstablished. Then server sends CMsgSteamDatagramConnectOK to client, After receiving - client swaps CMsgSteamDatagramConnectionStatsClientToRouter with server's CMsgSteamDatagramConnectionStatsRouterToClient many times (seq_num_c2r = 96 (raw) or 48 (zigzag) for me). After swaps - client closes old one with CMsgSteamDatagramConnectionClosed, creates new connection (probably to select primary server), and repeats everything before except ConnectionStats swapping and disconnect, after that it sends some data that's not similar to protobuf for me.

moofMonkey commented 5 years ago

@azuisleet I've made much better code, and now I'm stuck at CMsgSteamDatagramConnectRequest, which you've investigated in https://github.com/SteamRE/SteamKit/issues/123#issuecomment-538790122. To be precise - at method ConnectRequest. Can you help? I think client sends GameNetworkingSockets data (https://github.com/ValveSoftware/GameNetworkingSockets/blob/7125be2f552b5cfd421e5f53d32a0f08c86e0147/src/steamnetworkingsockets/clientlib/steamnetworkingsockets_udp.cpp#L1122-L1126) right after ConnectionOK message, so it may be one of last messages on this issue.

azuisleet commented 5 years ago

@MoofMonkey SDR requires modifications to GNS to make it compatible, as the only implementation is UDP currently. I'm not interested in taking on a project in a different language than SteamKit, so it would have to be something where a pure implementation of GNS is created.

paralin commented 5 years ago

@MoofMonkey Good work so far on this.

moofMonkey commented 5 years ago

@paralin can we contact at discord? I don't think I want to post any further RE since copypastas may blow Valve's servers. My discord can be found at my GitHub profile. https://github.com/SteamRE/SteamKit/issues/735 also @LWSS sent me this: https://www.unknowncheats.me/forum/counterstrike-global-offensive/357671-steamkit-clientless-bot.html

paralin commented 5 years ago

I agree, this is an area that rapidly expands the potential for abuse. Let's tread carefully.

moofMonkey commented 5 years ago

image Cool.

moofMonkey commented 5 years ago

@AlexanderBrese by specifying "clientless bot" you exactly suit "may introduce many CS:GO and Dota leveling bots", but not "many sites/companies will be able to spectate games without wasting human time and make people closer to better custom games hosting (better API, performance)", also you've posted a thread about this on UC without even trying to dig anything except enums by yourself. https://github.com/SteamRE/SteamKit/issues/123#issuecomment-538755597

moofMonkey commented 5 years ago

@paralin @azuisleet can we create conversation somewhere like discord? E-Mail isn't preferred. Here's invite to discord server. I've got crypto stuff working (it calculates keys and IVs for both peers), but I can't figure out for 100% what's packet structure for SDR. Going to share additional details and sources, of course, but not publicly.

moofMonkey commented 5 years ago

@paralin is that you? paralin#36xx Looks like yes because of connected accounts

yaakov-h commented 4 years ago

Closing due to inactivity.

If anybody wants to seriously pursue this as a SteamKit integration, please open a new issue or drop into IRC for discussion.