Replace RMI

[RoiEXLab]

As you may or may not know, tripleA uses the Java RMI system, which stands for Remote Method Invocation for it's internet communication... I have a more or less concrete idea on how to implement a Network-Communication-System using Java Sockets, but I'm here to discuss the details with you since this is going to be a huge project... I have a Packet System in mind - see my current implementation for more details. This Packet systems uses IDs for every action. Currently I have:

• HANDSHAKE = 0 || Sends client tripleaversion, username and password, password is used for lobby games only (otherwhise left blank)
• MOVE = 1 || Sends a unique destination territory id, a unique target territory id, and a unique unit id
• MOUNT = 2 || Sends a unique destination territory id, a unique target territory id, a unique unit id, and the unique unit id of the unit it wants to mount on (e.g. If a Tank wants to be shipped by a transporter etc.)

Is there anything missing? How are we going to restart the lobby without killing the java process?

[RoiEXLab]

@DanVanAtta thoughts?

[ron-murhammer]

I think you have the right idea here. This would be a very large undertaking though. You'd probably want to draw up a pretty full design before doing much coding.

I'd also wonder is there a more standardized format/protocol? I want to avoid custom code as much as possible.

[RoiEXLab]

@ron-murhammer Well using sockets, java is already helping us a lot, building a fully encrypted tcp connection to the server... Also, my attemp isn't too custom; it's just an id, a byte array length, and the payload containing custom data.

[ron-murhammer]

@RoiEXLab I guess sockets and a custom format payload seems kind of outdated. I do mostly web development these days so I'm used to thinking HTTP with JSON/XML/YAML. I realize this is different since we have a desktop client but wondering whether there are socket alternatives and if we should try to move towards a more standard format of JSON/XML/YAML. The 'sendPacket' method just makes me cringe as it would seem as this point there would be some framework to abstract a lot of that.

[RoiEXLab]

@ron-murhammer Ahh that's what you mean... Of course we could send JSON data, but the question I'm asking myself is, if this is taking to much server load, since every "packet" needs to be turned into json and backwards again. Also json is bigger than binary data. But sure, let's discuss this here

[ron-murhammer]

@RoiEXLab I wonder if something like protobuf would be a good solution: https://developers.google.com/protocol-buffers/

I'd be interested in determining what other online turn based strategy games use.

[RoiEXLab]

The protocol buffer could be a good way to store savegames in the future! It could be used to send data over the network, but would still use sockets and a bytestream containing too much data. If we send an object to the server the server can't rely on this object if it contains too much information about the game e.g. If we send a territory the server should validate this object and then respond to it

[DanVanAtta]

Protobuf can have some coupling/generation issues. I've worked with it before and was not very impressed.

The general idea though is right, we need something to marshal/unmarshal objects over network. I'm in agreement with looking to JSON over HTTP. Jackson has a very nice JSON<->Object mapper, and GSON by google is a good format with a lot of library support. For example, looks like it can be streamed pretty easily: https://sites.google.com/site/gson/streaming, would just need to connect that to a network socket for a proof of concept.

One thing to consider as well is if we can do things incrementally.

[RoiEXLab]

@DanVanAtta gson looks very promising, although I'm still convinced my attempt is the most resource saving, network wise. As I already mentioned with the protocol buffer gson could also be a good way to store a savegame. If we gzip and minify the json file, the filesize could also be reduced

[DanVanAtta]

Network payload size is not the most important consideration yet IMO. Chances are anything we switch to will be smaller than the object graphs that RMI is sending.

We need to keep in mind the transition/migration path in the current code base. A lot of the RMI is done via the Change event objects. I see a migration plan needing to first:

• survey where we do RMI. Where do we send objects over network, and which ones we do we send?

If we then can come up with something that handles nicely the worst case and a couple of other example ones, then with some luck we'll have a pretty clear migration path.

[simon33-2]

Curious suggestion. Why does this result in better outcomes for users?

[DanVanAtta]

@simon33-2 we'd be able to update the game engine and lobby independently and without the incompatibility surprises. It's exactly to avoid things like the sudden bot crashes that have been happening of late.

[RoiEXLab]

Exactly! Not 100% independently but better than currently

[RoiEXLab]

Whats the current state on this? I believe it would better if we were not going for an RESTful API. IMO this would just produce unnecessary overhead.

[RoiEXLab]

While reading this again: Of course there's something missing, a lot actually. We'd need Phase change packets, start battle packets, dice result packets, etc., etc., etc.

[ssoloff]

@RoiEXLab Offering some thoughts as requested... :smiley:

Let me clarify what I meant when I said I was planning on prototyping something related to this. I see several distinct problems that need to be addressed (and I think I'm echoing what you, @DanVanAtta, and @ron-murhammer have already stated here and in other issues):

1. Decoupling the domain model from the persistence model
2. Defining a new persistence model
3. Defining a new network interface

Decoupling the domain model from the persistence model

This is what I want to work on first because I'm experiencing how painful it is to be able to do even simple refactorings in the domain model without worrying about breaking backwards compatibility. I've actually done this on two projects in the past (using something very much akin to the Serialization Proxy pattern that has already been discussed), so I think I know the path to take here.

To stay focused on the decoupling, I plan to keep Java serialization as the default persistence model and not be concerned about item (2). Overall, it will require more work to do items (1) and (2) in two distinct steps. They could be done at once, but I'll have enough trouble keeping item (1) straight in my head on such a large codebase. :smiley:

Defining a new persistence model

If I'm not mistaken, the domain model today is persisted using Java serialization in all contexts (i.e. save games, network, etc.) due to the coupling of serialization to the domain model. Item (1) would allow for the flexibility to use either a single persistence model or multiple persistence models across all contexts. For example, a verbose textual format (that could be subsequently compressed) for save games to allow easy hand editing, and a tight binary format for the network.

I tend to agree with the opinions expressed above that JSON is probably a good start if a textual format is desired. @DanVanAtta already pointed out the benefits that existing JSON libraries provide. However, I don't have a good feel for how much data is actually transmitted during network games. So I'm not sure how important it is to save every possible byte in the persistence model representation. I'm hoping this isn't a huge concern for TripleA, as it's much more pleasant to have a persistence format that is easily consumable by humans when it comes time to debug without having to write tools to parse the data. 😃

Defining a new network interface

This is necessarily coupled to item (2) because, presumably, the payload of the protocol you choose is generated via a distinct persistence model (when that payload consists of domain objects).

As I mentioned in #1613, I would like to see a web client for TripleA. That biases my opinion that the new API should be easily accessible by web clients, but definitely by non-JVM clients, in general. My vote would be for something HTTP-based simply because every platform has access to an HTTP client. However, it doesn't necessarily have to be RESTful. Using REST, RPC, or some hybrid of the two would probably work.

Again, I don't have a good feel for the performance implications of TripleA, so I can't address earlier concerns about the overhead of certain protocols. It might take prototyping a few endpoints and then measuring various metrics to give a :+1: or :-1: on any particular proposal.

@DanVanAtta proposed above surveying the existing network interface and publishing that information in one place (here? wiki?). I think that's a great idea and would help to reason about the pros and cons of choosing specific protocols.

[RoiEXLab]

@ssoloff I did some research on what can be done in order to be "web-compatible", in other words be accessible using javascript. In my opinion, HTTP alone is not suited for that kind of task. HTTP doesn't allow for "real" server transmission of data unless explicitly requested... Our solution are websockets. Basically a mixture between HTTP and a regular TCP Socket. I found a simple java implementation, which isn't that much different from my code. A little bit more request handling, but that's it... WebSockets are supported in every common browser (even in IE 10). Read more here From german wikipedia, a sample js implementation:

var socket = new WebSocket(urlToWebsocketServer);

// callback-Function is called when the connection was established successfully
socket.onopen = function () {
    console.log("Connection was established successfully");
};
// callback-Function is called when a new websocket message is recieved
socket.onmessage = function (messageEvent) {
    console.log(messageEvent.data);
};
// callback-Function is called if an error occurs
socket.onerror = function (errorEvent) {
    console.log("Error! Die Verbindung wurde unerwartet geschlossen");
};
socket.onclose = function (closeEvent) {
    console.log('The Connection was closed --- Code: ' + closeEvent.code + ' --- Reason: ' + closeEvent.reason);
};

[ssoloff]

@RoiEXLab :+1: Yes, web sockets would be perfectly acceptable from my perspective as they are a (now) common web client technology. I didn't even think to ask how much TripleA network traffic is pushed by the server, but your statement above makes it clear that it's common enough that old school HTTP polling would not be acceptable. :smiley:

I don't have much experience with this on the Java side, but many Node-based service frameworks encourage you to write your services independent of the protocol, and then provide adapters so that their endpoints can be exposed using REST, web sockets, message bus, etc. That might be something to keep in mind as the networking code is redesigned. It may be that one-size-does-not-fit-all for any single protocol, and some services are better exposed with another protocol (e.g. web sockets for chat but REST for downloading maps).

Keeping the services that enable distributed play ignorant of the actual protocol just seems like a Good Thing. :smiley:

[RoiEXLab]

Just found out that Java has a websocket client as part of it's standard library. We can send data in a binary format, not sure if this is better than using JSON

[RoiEXLab]

Since we probably want to keep up with web standards, and therefore have a secure connection which encrypts passwords etc. we'd need a (sub-)domain pointing at the lobby server in order to generate a valid certificate using let's encrypt. I'd recommend something like lobby.triplea-game.org in the furture @DanVanAtta .

But I agree with @ssoloff First step would be to separate server and non server code...

[RoiEXLab]

There's also a library which adds in WebSocket support with SSL/TLS... https://github.com/TooTallNate/Java-WebSocket Leaving this here, for later usage https://github.com/certbot/certbot/issues/1701#issuecomment-163986593

[DanVanAtta]

Sorry to not get full context on this convo, so this is a bit of a drive-by comment.

What I think we need is simply a good way to send objects back and forth. EG: we can tell the game to send an object, and on the other side we read an object. The current RMI does that very literally and directly. We need a layer of software in between that introduces a sane inter-change format so we can control that.

So basically we need a library that can take a tripleA object, and convert it to an object that we can send over network, and then on the other side we need something that can read and convert up to an object. In json, this would look like a standard toJson method and fromJson method on objects that we want to send and receive over network.

The migration path is pretty kind if we take that approach even. We can still for better or worse re-use the existing game infrastructure, just we would send the and read the objects in a new manner.

[ssoloff]

@DanVanAtta

So basically we need a library that can take a tripleA object, and convert it to an object that we can send over network, and then on the other side we need something that can read and convert up to an object. In json, this would look like a standard toJson method and fromJson method on objects that we want to send and receive over network.

:+1: We're on the same page here. I think you'll see this is the case in the prototype I intend to submit. Now I just need time... 😃

[RoiEXLab]

I'm not sure if turning java objects automatically into json is the right way, since this probably introduces the same problem we have with serialization. We can't change the variable names without losing backwards compatibility.

[RoiEXLab]

Also when sending objects automatically, we might end up sending too much data. I'd personally prefer a manual way. Where we choose the needed data individually

[ssoloff]

@RoiEXLab In the solution I'm thinking of, it won't happen "automatically." Think of it more like having an adapter that converts between DomainObject and its JSON (or whatever) representation. The adapter would only use the public API of DomainObject to do the conversion. The adapter would be coded manually and can be refactored along with DomainObject. Presumably, the adapter would also handle versioning issues (e.g. it would know how to convert the previous version's JSON representation to the current version, if necessary).

[RoiEXLab]

Good, I just thought @DanVanAtta was aiming for an automated solution.

[ssoloff]

@RoiEXLab I should amend my previous statement that the framework I'm envisioning would not preclude automatic conversion. One could register an adapter for a specific DomainObject that used reflection or some other heuristic to automatically marshal/unmarshal the object. It's just that I've found (as you implied) that automatic conversions could lead to the same problems we're experiencing today.

[DanVanAtta]

@ssoloff @RoiEXLab Some interesting items in this list: https://github.com/akullpp/awesome-java#data-structures, particularly some interesting things like: https://github.com/square/wire

We could perhaps use a listing of things we'd like, requirements. From my point of view, the primary is something where we can update our java classes and allow for the updates to be backwards compatible to network clients (IE: clients ignore unknown properties).

Secondary requirements:

• simple, ideally very little code overhead to marshall / unmarshall

[DanVanAtta]

I'm having a lot of luck with some quick experimentation with DropWizard. It did not take much to set up a simple server: https://github.com/DanVanAtta/triplea/blob/dropwizard_experiment/src/main/java/org/triplea/lobby/LobbyApplication.java

https://github.com/DanVanAtta/triplea/blob/dropwizard_experiment/src/main/java/org/triplea/lobby/challenges/ChallengeService.java

A simple client was pretty straight forward as well: https://github.com/DanVanAtta/triplea/blob/dropwizard_experiment/src/main/java/org/triplea/client/dropclient/LobbyClient.java

---

I'm working on a PBEM challenge UI and server components, it seems to make sense to build that directly in a new technology. First, the client UI needs some updates to make adding a challenge panel easier. I plan to work on that first and then return to the server/client PBEM challenge system.

[ssoloff]

Looks neat, @DanVanAtta!

When you get the first service into the code, I'd like to test a couple of the endpoints with a totally different (i.e. non-DropWizard-generated, non-Java) client. I've still been tinkering with a web client, and I want to ensure there's no surprises on the wire when the other side doesn't use the exact same technologies. I didn't see anything in your prototype code that would suggest this might occur, but just being overly cautious, especially where JSON/Object mappers are concerned.

[DanVanAtta]

FWIW, before adding the sample java client, I used curl to make sure I could get a JSON back. I'm pretty confident any client would work. Adding a keystore with a cert should not be too terrible either and we can then use https.

[ssoloff]

Did you notice any unexpected "underscore" properties in the JSON? Or were they all a straightforward mapping of the object you were sending back in the response? I can't say I've noticed this lately, but I remember in the past, some mappers would assume they were the only ones marshalling/unmarshalling, so they added extra metadata to the JSON to help them do it. 😞

[DanVanAtta]

The response seems to be very 1:1 with what you would expect. The @JsonProperty annotation allows for the name to be customized so you are not stuck with the variable name.

When interpreting requests we can turn on automatic snake to camel case conversion.

My experience with drop wizard has been positive previously. For example, a service endpoint was pretty easy to configure, and notice we return an immutable value object that is automatically converted to JSON for us:

@Path("/availableGames")
@Produces(MediaType.APPLICATION_JSON)
public class ChallengeService {
  @GET
  public List<MatchChallenge> getGameListing() {
    return Arrays.asList(new MatchChallenge());
  }
}

Here are the properties defined in the value class:

  @JsonProperty
  private final String value = "abc";

  @JsonProperty
  private final String valueCamel = "valueCamel";

  @JsonProperty
  private final String value_snake = "value_snake";

  private final String hidden = "hidden";

Curl output, pretty no-thrills JSON, hidden was not annotated so it is not included. I think it's possible to do complex types and do composition.

dan@dan-desk:~/work/triplea$ curl localhost:8080/availableGames
[{"value":"abc","valueCamel":"valueCamel","value_snake":"value_snake"}]

[ssoloff]

:+1: Cool, looks like any modern JAX-RS/Jackson stack. Just didn't want to see a property in the JSON-encoded form like "_dwType": "games.strategy.MyValueClass" or something. :smile:

[RoiEXLab]

Looks good to me! Keep it up! I actually finished a dice server software using Node.js, but the files were corrupted due to a power outage, so I may not be able to recover them. In any case (even if I need to rewrite them which shouldn't be too difficult because I know what I wrote already) we could use this software as a lightweight dice server, we'd just need to support this kind of software with triplea (simple json handling). The question is if we should actually have a separate dice server or we could just use the lobby code to serve as verifier without the need for a separate software.

[DanVanAtta]

Thanks! Re: dice server. I think a driving question is what infrastructure you would like to re-use. A simple stand-alone dice server seems fine. An auth piece probably is needed somewhere, smtp servers are known to get hijacked. I'm reluctant to see more code be added to the lobby as is

[RoiEXLab]

I was thinking about dropping the whole email notification thing to be honest, I never really played PBEM, but I find the email notifications quite annoying. The solution would be to store the dice signature along with the savegame (if it isn't done already) and then request the servers public key to verify the result /send the signature to the server and let the server check the signature. This way we wouldn't need any sort of authentification

[DanVanAtta]

I think the intent was to make the dice server usable for a tourney where some players are more incentivized to cheat. Would an ordinary player be able to follow those steps to verify dice?

[RoiEXLab]

With dice signatures embedded in the savegame, triplea could automatically verify dice rolls

[DanVanAtta]

That is way nicer than email :+1:

[prastle]

PBEM is rough but dan is correct about why they wanted pbem. To elim cheats. I fully vote my 2 cents for a non pbem @RoiEXLab

[prastle]

hmm how do i upvote here ;)

[simon33-2]

There is also recovery from crashes for which email is useful. Please consider this aspect!

Upvote is possible with the smiley face in the top right.

[prastle]

Ty simon

[prastle]

i dont think anyone is saying remove pbem.

[prastle]

it just gets done elsewhere i would hope

a separate marti or dice server is a good idea imho

[RoiEXLab]

@DanVanAtta Thinking again about your dropwizard project: While dropwizard might be a good way to achieve your examples, it might not work out for actual game hosting due to the restrictions of HTTP 1.1 Fortunately Dropwizard supports HTTP2 and server side pushes (which requires HTTPS in case you didn't know) which would just be the missing functionality we'd need. (We'd also need to update to the alpha version of httpcomponents to get client side HTTP2 support but that should be a minor problem.) There's just this problem with P2P game connections... We can't require users to get certificates on their own just to have HTTPS work, so we'd need to figure out a way to get that working, likely by "ignoring" a self signed certificate in such cases. Thoughts?💭

[DanVanAtta]

Thanks for the interest and follow up @RoiEXLab . I was meaning to post similar. For lobby hosting, it perhaps is a very good choice.

Game hosting on the other hand would require some re-work. To fake not having pushes, clients would have to requset updates periodically. The game is not quite set up for that, would be an effort.

Though, looking at this, it does seem we can change the output stream serializer from an object output stream to a custom output stream writer. Then we'd only need that to write JSON, and create a framework to encode object messages and serialize them back to objects on the other side.

[DanVanAtta]

Closing for now, we now pretty well we want to replace RMI. Specific suggestions are likely best taken up as new issues.