Support HLL serialization as protobuf directly usable in Google products (e.g. BigQuery)

constellation-rs / amadeus

Harmonious distributed data analysis in Rust.

https://constellation.rs/amadeus

Apache License 2.0

472 stars 26 forks source link

Support HLL serialization as protobuf directly usable in Google products (e.g. BigQuery) #134

Closed lquerel closed 3 years ago

lquerel commented 3 years ago

Added a new function serialize_as_proto returning a serialized representation of the HHL directly usable by BigQuery.

See Zetasketch from Google for more details on the serialization format.

See HLL functions in BigQuery to understand how to store and use the sketches generated by this function.

lquerel commented 3 years ago

Sorry for this late answer, I was a little busy those days. Submitted a set of fixes based on your review. Let me know if there is anything else to change.

alecmocatta commented 3 years ago

It seems quite common in rust crates for e.g. serde integration to be enabled by a feature named "serde", so I renamed the feature to adhere to that convention here. Thanks for the PR @lquerel!