search

influxdata / influxdb

Scalable datastore for metrics, events, and real-time analytics
https://influxdata.com
Apache License 2.0
28.71k stars 3.54k forks source link

Create a type for serializing Arrow schema to the catalog #25031

Closed hiltontj closed 3 months ago

hiltontj commented 4 months ago

This issue is tracking the addition of a serialization layer that converts Arrow Schema to JSON for the catalog.

The types for doing so can reflect Arrow's Schema type and its descendants, but will implement serde's Serialize/Deserialize.

Although Arrow's types support serde's traits, the types themselves are not stable, so we are adding this layer to guard from potential breaking changes from the arrow ecosystem.

hiltontj commented 4 months ago

Part of the purpose of this effort is to remove the dependency on the core data_types::ColumnType to represent column types in our catalog, and rely more closely on Arrow's DataType.