Store Protobuf messages in ArangoDB

[jevontech]

I would like to be able to store Protobuf Go structs directly in ArangoDB. The problem is that the standard "encoding/json" marshall/unmarshall doesn't work correctly on all Protobuf messages. For example timestamps are not correctly encoded/decoded. The protobuf project provides their own JSON marshaller in "google.golang.org/protobuf/encoding/protojson". Unfortunately the ArangoDB go-driver only supports "encoding/json".

I see two options to solve this problem:

• Either create a duplicate for all methods that currently have a "document interface{}" as input or output that operates on raw JSON (byte[]) so the user can do the JSON marshall/unmarshall step. Probably not ideal as this significantly increases the API interface.
• Add an configuration option to use protojson as JSON marshall/unmarshall instead.

Do you see any other ways to solve this problem ?

I would be willing to help with the implementation.

Many databases don't work well with Protobuf messages due to similar issues and because ArangoDB already uses JSON as their internal data format , it could be an unique selling point for ArangoDB that Protobuf messages can be stored directly (without copying data from the Protobuf struct to a regular Go struct in order to store it in a database )..

[nashikb]

AFAICT, technically AQL uses velocypack, and while the API can be configured for velocypack, there's a hard-coded override in the code.

A workaround to an interim marshaling step could be to wrap the PBs with a type that has a custom MarshalJSON interface in cases where timestamps and durations are listed. Part of this is that PBs timestamp/durations are... novel.

But I could see protojson added as an encoding option to both AQL and API. I haven't looked deeply at the code, but this could be complicated in the AQL case if a need would arise to mix the marshaling of different bindvars, for example.

This might be a good v2 feature if Team Arango would be willing to take the contribution?

[jwierzbo]

Right now, we can't do this.

The default for the server is to receive and send JSON data. Many drivers do not implement velocypack and are thus relying on the JSON APIs. That also means we cannot add support for any data types that cannot accurately/losslessly be mapped to JSON types. If we did anyway, then converting such types to JSON would either fail or produce data loss.