Support for already serialized documents

[massimiliano-mantione]

I have an issue very similar to #217.

I already have a json [de]serialization framework in place, which is not the newtonsoft one and is not compatible with anything else, except that is is very basic and works with Dictionary<string,object>, using Boolean, Double, Int64 and UInt64 instances instead of strings where appropriate.

I wanted to use rethinkdb mostly for the query-change-push feature.

I could try to use the existing Dictionary<string,object> support in this driver but I am not sure that it would play nice with the framework I already have. I tried looking at the source code but got a bit lost... how hard would it be to just bypass [de]serialization completely? Or, alternatively: are there a couple of utility methods to [de]serialize Dictionary<string,object> to-from strings "the rethinkdb-net way"? It would be a bit inefficient but I could live with it.

Thanks!

[mfenniak]

Hi Massimiliano. I'm sorry to say that it is not possible to use an already serialized document.

rethinkdb-net needs to follow the RethinkDB network protocol. They have adapted the protocol relatively recently to use JSON as a serialization, but, it's not the documents / data themselves that are serialized into JSON. The protocol commands are serialized into JSON in a funky mechanism. So, rethinkdb-net requires that all documents make their way into the RethinkDb.Spec.Datum format (in https://github.com/mfenniak/rethinkdb-net/blob/bd0215fa9c3d13731bf74bb09d2eca6ae4b5a9e1/rethinkdb-net/rethinkdb_spec.cs), which the driver then knows how to communicate with the server. You can see the format that rethinkdb-net converts Datum objects to JSON as for the network communication here: https://github.com/mfenniak/rethinkdb-net/blob/bd0215fa9c3d13731bf74bb09d2eca6ae4b5a9e1/rethinkdb-net/Protocols/Version_0_3_JsonProtocol.cs

In order to do this, we rely on an interface called IDatumConverter. Basically each instance of an IDatumConverter converts a specific type to a RethinkDb.Spec.Datum object. These IDatumConverter instances are wrapped into a higher-level IDatumConverterFactory instance, which knows how to provide IDatumConverter objects for a specific type. These are all pretty easy to plug-in custom implementations to a Connection.

Another alternative approach might be for rethinkdb-net to support the RethinkDB JSON command (https://www.rethinkdb.com/api/javascript/#json). This command sends an entire document as a string from the client to the server, and the server parses it into the object representation. The problem is that I haven't figured out a way to use the JSON command and maintain any of the type-safety aspects that rethinkdb-net enjoys... so I'm not sure then what querying or updating documents would look like in terms of rethinkdb-net's API.

[massimiliano-mantione]

Hi! I was reading the spec file and I was hoping that I could use the R_JSON encoding, but then I sow that in rethinkdb-net there is no code to use it.

However, a "custom" IDatumConverter that gives me an untyped but deterministic dynamic structure would be ok for my needs. A combination of Dictionary<string,object>, List

[mfenniak]

The DatumConverter for Dictionary objects is located here: https://github.com/mfenniak/rethinkdb-net/blob/4cae29bf6eedbb5099954acab7cd2eeb5bda44e3/rethinkdb-net/DatumConverters/NamedValueDictionaryDatumConverterFactory.cs.cs

(It's called a NamedValueDictionary... because it only supports Dictionary objects where the keys are strings, just like a JSON object, but any object can be a value.)

There are also datum converters in the same file for the types Dictionary<,>.KeyCollection and Dictionary<,>.ValueCollection, which you can probably ignore.

[massimiliano-mantione]

Getting there :-)

I noticed that every dictionary value will have its converter picked by GetBestNativeTypeForDatum, and I presume that the concrete implementation is the one in AbstractDatumConverterFactory. (is this right?)

If it is so, I see that heterogeneous arrays are not supported. What I really need is a converter that will create List<object> for arrays.

Actually, what I probably need is an implementation of a converter with a switch on Datum.DatumType that will always return what I need (as I wrote above, Dictionary<string,object>, List<object>, string, Boolean, Double, Int64 and UInt64). And a converter factory that always, unconditionally returns that converter.

Again, finding my way through the code is a bit hard, but... would it be possible to write those two classes (the factory and the converter) outside the driver, and pass them to the driver top level API so that I get exactly the conversion that I need without rebuilding the driver?

If yes, could you please give me a few hints about how to do it? Like a dumb converter (and its dumb factory) that always creates a string calling ToString(), and sample code on how to use it in a query?

Given that converter I could use it as a skeleton to write the one I need. I would then contribute it back, of course :-) I am just not sure that it would have a place in the driver because everything seems focused on strongly typing every value, while in this case what I need is exactly the opposite.

[mfenniak]

Again, finding my way through the code is a bit hard, but... would it be possible to write those two classes (the factory and the converter) outside the driver, and pass them to the driver top level API so that I get exactly the conversion that I need without rebuilding the driver?

Yes, the Connection object has a QueryConverter property on it that describes how the driver will handle converting basically everything in .NET to RethinkDB datums and terms. You can see the default object being initialized here, and it includes all of the default datum converters. You can start by copying this code, and strip out of any of the converters you don't need, and add your own. Then just set it on your connection object.