Open NickCondron opened 1 month ago
Thanks for raising this. I looked into it a while back, and determined that it was going to be a nontrivial amount of work for no immediate gain. I guess we'll eventually have to switch, but I'm just not very motivated to slog through it at the moment.
Maybe the first step would be to note the salient differences between the two libraries. Help would be welcome there.
I took an initial crack at it. Here are notable differences so far:
MutablePrimitiveArray<i32>
becomes PrimitiveBuilder<Int32Type>
PrimitiveArray<u32>
becomes PrimitiveArray<UInt32Type>
NullBufferBuilder
type. It finalizes to Option<NullBuffer>
only allocating the buffer if there is at least one null. This simplifies some of the repetitive logic for this that is currently in peppi.Offsets
array (called OffsetBuffer
in arrow-rs)I haven't done much, but what I've done so far might help illustrate some of these: https://github.com/hohav/peppi/commit/ef50bf2d214e7e5a9b46aa655ea8a8efb99dfbbb
https://github.com/jorgecarleitao/arrow2/issues/1429