Go JSON / Binary / String support

[mattheworiordan]

As discussed with @kouno, we need to be aware that ALL of our client libraries natively support three payload data types namely:

• String UTF-8
• Byte arrays (binary)
• JSON

Typically in dynamically typed languages, when you subscribe to messages, you will be delivered the data type automatically because when the message is received, it is decoded automatically, see https://github.com/ably/ably-ruby/blob/master/SPEC.md#ablymodelsmessageencoders.

In more strongly typed languages such as Java, we still encode & decode automatically and return the correct data type, see https://github.com/ably/ably-java/blob/db50628fceb0af37c63c33693ab77dbb35d3c245/src/io/ably/types/BaseMessage.java#L97-L135.

However, in Go, JSON is not a common data type and instead the norm is to Marshal JSON strings into typed Structs. The problem for us is that whilst we return a JSON data type in Ruby / Javascript / Java, for us to do this in Go we'd need to return a generic {{Interface{}}} object, see http://golang.org/src/encoding/json/decode.go?s=2340:2388#L57. We could do this, but my understanding is that this is largely frowned upon the Go community and instead JSON should be marshalled to a Struct. If we did return a generic interface, it's quite probable the developer would only then need to marshal it into another object so we're just adding unnecessary processing.

I don't have a solution, but am raising this issue so that we can discuss.

@paddybyers any thoughts?

[mattheworiordan]

As an aside, in our documentation I think from a performance perspective we should be encouraging users to pack their objects with MsgPack and send as binary payloads. If they do this over a binary protocol the process is as follows:

Object => MsgPack => Ably Client => Ably => Ably Client => MsgPack => Object

If instead they used JSON for example, it would be more like this:

Object => JSON => MsgPack => Ably Client => Ably => Ably Client => MsgPack => JSON => Object

[rjeczalik]

We could do this, but my understanding is that this is largely frowned upon the Go community and instead JSON should be marshalled to a Struct.

Java's JSONArray ~is Go's []interface{}, the same JSONObject -> map[string]interface{}. In order to just have the functionality of automatic decoding, the interface{} is fine.

Even if either:

• user provided a TypeMap (map[string]reflect.Type) for each channel to instruct the Channel how to unmarshal the payload for each message before delivering it to the listener
• we built the type map information automatically by inspecting user-provided listener with reflection
• we wanted to defer unmarshmaling of the payload to the user by making Channel configurable, so the user could then use raw []byte and unmarshal it by hand into a struct

there'd be an limitation that user would have to ensure the payload type is the same for each message with the same name for the given channel. In other words if user is able to map protocol.Message programatically to his/her own struct, then we're able to do it automatically. I don't see how that mapping would look like.

So we're left with the default behaviour of unmarshalling into interface{} - that is to unmarshal payload to generic interface{} (either an array - []interface{}, or an object - map[string]interface{}).

[mattheworiordan]

@rjeczalik is there no way that we could return an []interface{} object if they simply access the payload, but perhaps offer an alternative where they can pass in the Type and we will marshal it for them based on that type?

Also, our payload needs to support binary, strings & JSON. How do you recommend we approach that given we typically have one method such as message.data to access the data payload? I understood overloading is not possible in Go.

[paddybyers]

instead JSON should be marshalled to a Struct

I agree with @rjeczalik that that would only be possible if there was a schema given for each possible message data type, indexed by message name. We did discuss this at one time for the java lib also but we decided it's not appropriate at this time, and probably not appropriate at all.

Assuming that the library de-serialises by default to a generic interface, then we might consider an option that suppresses the default de-serialisation, passing the raw buffer (and encoding string) back to the client to de-serialise.

[paddybyers]

I think from a performance perspective we should be encouraging users to pack their objects with MsgPack and send as binary payloads.

I don't get this comment. If they pack with msgpack they give us a buffer, and if they pack with JSON they give us a string. In either case, that native type undergoes no further processing when we pack it into the ProtocolMessage msgpack. (Of course if they are using the text protocol then the string is much better because it avoids the base46 step.) The only other difference is if they are using encryption, in which case having them supply a biinary payload removes the utf8 encoding step.

[mattheworiordan]

@paddybyers my point is that if clients use JSON it's very likely they JSON data came from an Object of some sort. So in reality (and I've seen this when reviewing Aqueduct's implementation) they will probably serialise their Object into JSON, pass that to the client library, which will then serialise that into MsgPack as part of the ProtocolMessage, and all of this will happen again on the other end. If however a client developer uses MsgPack, the cycle is vastly better, there is one serialisation from Object to a MsgPack byte array which when passed to the client library will not need to be encoded, it's simply embedded into the ProtocolMessage MsgPack object. Then on other end again, there is on conversion back to MsgPack but the payload is left as a byte array (again no processing) which in turn is serialised directly back into the Object the developer will be using in a far more efficient manner.

My point is that we should encourage people to serialise & deserialise their objects with a binary format such as MsgPack, BSON, Thrift etc and pass us this payload in binary format as that is certainly the most efficient.

[rjeczalik]

@rjeczalik is there no way that we could return an []interface{} object if they simply access the payload, but perhaps offer an alternative where they can pass in the Type and we will marshal it for them based on that type?

If user provides a type then it's possible. But this is exactly the problem I described - if user is able to map the massage to a type, then he/she would also be able to provide a type mapping and make the payloads be unmarshalled automatically.

Also, our payload needs to support binary, strings & JSON. How do you recommend we approach that given we typically have one method such as message.data to access the data payload? I understood overloading is not possible in Go.

interface{} is the best what we can get. Even if we're able to extract the payload type from the Encoding field - json/ciphers for JSON, utf8/ciphers for string and rest for binary, we would need to provide some kind of enum to designate the kind of payload, which will be no different than plain old type switching:

switch payload := message.Data.(type) {
case []interface{}:
    // handle JSON/MsgPack array
case map[string]interface{}:
    // handle JSON/MsgPack object
case string:
    // handle UTF8 string
case []byte:
    // handle binary
}

[mattheworiordan]

Ok, thanks for the explanation @rjeczalik

[mattheworiordan]

Is this resolved @rjeczalik?

[rjeczalik]

@mattheworiordan No, automatic payload decoding is still missing.

[tcard]

This is done now.