Currently we are storing the provided JSON data blob in the client data store as is (potentially gzip compressed) but when it comes time to generate a telemetry report JSON payload we are unmarshaling the JSON telemetry data blobs into in-memory Go data structures to create in-memory telemetry bundles, which are included into an in-memory telemetry report, which is then marshaled as a JSON telemetry report request payload.
It should be possible to define custom marshal & unmarshal handing using MarshalJSON and UnmarshalJSON methods for the payload field of the in-memory telemetry data item data structure that will eliminate the unnecessary unmarshaling and re-marshaling of the provided JSON data blob.
We should be able to define a type to represent the telemetry data item's payload that implements MarshalJSON and UnmarshalJSON methods such that we just store the JSON blob value untouched as the data item payload when we are creating the in-memory representation of telemetry data items, telemetry bundles and telemetry reports, and then when we go to marshal the telemetry report to JSON the MarshalJSON handler for the telemetry data item payload type will just return the JSON data blob as the encoded representation, without touching it.
In essence, on the telemetry client side, the provided JSON blob should only ever be unmarshaled into a temporary variable when it is first received by the telemetry library as part of validating that it encodes a JSON object. There after we shouldn't be touching the original encoded representation except to potentially compress it as part of storing it in the client data store.
Similarly on the server side, when we receive a telemetry report request and unmarshal it's JSON payload, we shouldn't be touching the JSON data item representation before storing it in the Telemetry DB unless there is defined storage transform for that telemetry type.
Currently we are storing the provided JSON data blob in the client data store as is (potentially gzip compressed) but when it comes time to generate a telemetry report JSON payload we are unmarshaling the JSON telemetry data blobs into in-memory Go data structures to create in-memory telemetry bundles, which are included into an in-memory telemetry report, which is then marshaled as a JSON telemetry report request payload.
It should be possible to define custom marshal & unmarshal handing using MarshalJSON and UnmarshalJSON methods for the payload field of the in-memory telemetry data item data structure that will eliminate the unnecessary unmarshaling and re-marshaling of the provided JSON data blob.
We should be able to define a type to represent the telemetry data item's payload that implements MarshalJSON and UnmarshalJSON methods such that we just store the JSON blob value untouched as the data item payload when we are creating the in-memory representation of telemetry data items, telemetry bundles and telemetry reports, and then when we go to marshal the telemetry report to JSON the MarshalJSON handler for the telemetry data item payload type will just return the JSON data blob as the encoded representation, without touching it.
In essence, on the telemetry client side, the provided JSON blob should only ever be unmarshaled into a temporary variable when it is first received by the telemetry library as part of validating that it encodes a JSON object. There after we shouldn't be touching the original encoded representation except to potentially compress it as part of storing it in the client data store.
Similarly on the server side, when we receive a telemetry report request and unmarshal it's JSON payload, we shouldn't be touching the JSON data item representation before storing it in the Telemetry DB unless there is defined storage transform for that telemetry type.