y-scope / clp

Compressed Log Processor (CLP) is a free log management tool capable of compressing logs and searching the compressed logs without decompression.
https://yscope.com
Apache License 2.0
813 stars 68 forks source link

ffi: Add support for serializing a `KeyValuePairLogEvent` as a `nlohmann::json` object. #512

Closed LinZhihao-723 closed 3 weeks ago

LinZhihao-723 commented 1 month ago

Description

Note: this PR should not be merged before #507 and #511 This PR implements a method in KeyValuePairLogEvent to serialize it into a nlohamnn::json object. KeyValuePairLogEvent, as the output of Deserializer in key-value pair IR format, will be processed by the ffi libraries. Each ffi library has to turn this C++ object into some dynamic data structure in their language. Ideally, KeyValuePairLogEvent can be serialized into msgpack byte sequence. However, there are two problems we need to solve:

  1. In the current format, arrays are stored as an unstructured JSON string. We need to parse the JSON array to build the msgpack array.
  2. It isn't that straightforward to use the official C++ msgpack library to construct an in-memory map and then serialize it into bytes.

Therefore, we decided to serialize KeyValuePairLogEvent into nlohmann::json object instead for now. nlohmann::json provides a better support to construct an in-memory map, and also provides methods to serialize the result into either msgpack byte sequence or JSON string.

With this PR, we should have all the necessary pieces to integrate serialization/deserialization features of the key-value pair IR format into our ffi libraries.

Validation performed