PowerTAC communication extension proposal and discussion

[pascalwhoop]

Hello, I have looked extensively into different techniques for extending the capability of the powerTAC infrastructure to allow for better communication with various components written in different languages. I would like to summarize my findings which I have already discussed with John to receive feedback and opinions.

The problem

The communication as of today can be considered an internal API that is not specified or documented in any public ready way. This is due to the fact that historically, the comm has been handled by the same Java package on both sides. This is great, as it reduces complexity and avoids 3rd party devs to be concerned with it. But there are obvious downsides to it:

1. No clear spec of how the data looks like on the wire
2. No alternative clients existing

Change goals

• keep readability of communication alive (to facilitate debugging and trace logging)
• open comm up to 3rd party languages
• clearly specify message format
• allow automatic model generation (to avoid multiple manual implementations of the domain model classes)
• let new comm live side-by-side with old variant (so brokers can choose which channel they prefer)

Investigated alternatives

handleMessage 2 GRPC

The first (already implemented) approach is based on GRPC to transmit the messages between the Java sample-broker and the final client. For this, each handleMessage method in the three core classes of the sample-broker passes the received message along to the GRPC infrastructure. While previous developers have handled these messages in the Java environment, I pass these messages to the ultimate environment by converting them into protobuf messages which are then sent to a connected broker who implements correpsonding handler methods in the target language.

Pros:

• touching server not needed
• similar approach as previous sample-broker variant, as the callback handleMessage calls are just forwarded
• reuse of code that handles connection timeouts, login procedures, xml tagging etc
• protobuf messages have a toJSON method --> keep readability
• protobuf generates model classes from proto file as well as builder utilities

Cons:

• 3 step comm (server-adapter-client)
• Pojo -> xml -> Pojo -> protobuf -> Python

Expose XML messages via Socket on Sample-Broker

A second approach is quiet similar to the original bridge but instead of writing the XML strings to the local file system, they are passed to the fi- nal environment via GRPC by simple messages that just serve as a wrapper for the XML string. While this is not elegant from a engineering perspective (GRPC should be used on a method level and messages should not contain other message formats as strings), it is simple and may lead to quick results.

Pros:

• quick
• done

Cons:

• XStream XML becomes defacto API
• wrong use of GRPC
• reimplement model classes for each new target environment needed

Generate Schemas from Domain Model

A final approach is the generation of schema definitions from the Java model classes that are transmitted between the brokers and the server. JSON vs XML schema is the question. XML did not succeed for the PowerTAC model definitions which lead me to create a question on StackOverflow. The resulting answer lead to the ultimate alternative which is the generation of JSON schemas which can then be converted into Python class files .

pros:

• reuse of MQ infrastructure currently used
• simply switching over-the-wire representation of data
• JSON is more lightweight (on average)
• broader support from tooling, as JSON has largely replaced XML in APIs
• generation of model classes from formalized JSON schema message definition

cons:

• need to reimplement parallel concepts of XStream objectId -> reference resolution for JSON

[jecollins]

Another classical option is IDL (the CORBA Interface Definition Language).

[pascalwhoop]

I just found out that XStream crashes if there is certain "odd" structures of the XML stream.

Let's say we want to deserialize a Competition.

xml of the form

...
<simulationBaseTime>
  <iMillis>1277078400000</iMillis>
 </simulationBaseTime>
...

works. But

...
<simulationBaseTime>
  <iMillis> 1277078400000 </iMillis>
 </simulationBaseTime>
...

or

...
<simulationBaseTime>
 <iMillis>
   1277078400000
  </iMillis>
 </simulationBaseTime>
...

both fail. It can't handle the newlines or spaces, it's not stripping spaces/newlines etc That would make it extremely fragile to use XStream when opening the XML up to other XML creators

[pascalwhoop]

Interesting side note: Some performance comparison. Serializing the Competition object with all its content 1000x on my machine:

• Serialisation from Java to Protobuf is 44x faster (1619ms XML, 37ms protobufs)
• Deserialisation from Protobuf to Java 5x slower (7444ms protobufs, 1366ms XML)

Unsure why deserialisation is so much slower but I assume it's because I'm using some reflection to force the PowerTAC classes to take the ID that is incoming instead of generating a new one as well as some BrokerRepo etc lookups.