Introduce a generic Array / ArrayView abstraction that encapsulates the in-memory format in arrow independently from any concrete arrow implementation.
Motivation
Having a small, common abstraction layer makes it simpler to
Support different concrete arrow implementations (e.g. for #52)
Support different constructors of arrow arrays (serde or manual builders)
Design
Implement an Array / ArrayView Enum with variants for all supported data types. The arrays should be "open" for direct modification to allow construction outside of serde_arrow. The information should be included without redudancy (e.g., no separate length field for the validity bitmap and the overall array).
Store any field metadata that cannot be reconstructed from the array separately (name, nullable, metadata).
Steps
[x] Serialization for arrow (#197)
[x] Deserialization for arrow (#200)
[x] Field / Datatype conversion
[x] Everything for arrow2
Remove From<..> for ArrayDeserializer impls
Remove deserialization::construction submodule
[x] Refactor serializers / deserializes to the generic views / array directly
chmp
commented
1 month ago
Introduce a generic Array / ArrayView abstraction that encapsulates the in-memory format in arrow independently from any concrete arrow implementation.
Motivation
Having a small, common abstraction layer makes it simpler to
Design
Implement an
Array/ArrayViewEnum with variants for all supported data types. The arrays should be "open" for direct modification to allow construction outside ofserde_arrow. The information should be included without redudancy (e.g., no separate length field for the validity bitmap and the overall array).Store any field metadata that cannot be reconstructed from the array separately (name, nullable, metadata).
Steps
From<..> for ArrayDeserializerimplsdeserialization::constructionsubmodule