Switch from arrow2 to arrow-rs since arrow2 has been deprecated

NickCondron commented 1 month ago

https://github.com/jorgecarleitao/arrow2/issues/1429

hohav commented 1 month ago

Thanks for raising this. I looked into it a while back, and determined that it was going to be a nontrivial amount of work for no immediate gain. I guess we'll eventually have to switch, but I'm just not very motivated to slog through it at the moment.

Maybe the first step would be to note the salient differences between the two libraries. Help would be welcome there.

NickCondron commented 1 week ago

I took an initial crack at it. Here are notable differences so far:

MutablePrimitiveArray<i32> becomes PrimitiveBuilder<Int32Type>
PrimitiveArray<u32> becomes PrimitiveArray<UInt32Type>
When constructing your own validity you can use the NullBufferBuilder type. It finalizes to Option<NullBuffer> only allocating the buffer if there is at least one null. This simplifies some of the repetitive logic for this that is currently in peppi.
There's less guardrails and convenience for the Offsets array (called OffsetBuffer in arrow-rs)

I haven't done much, but what I've done so far might help illustrate some of these: https://github.com/hohav/peppi/commit/ef50bf2d214e7e5a9b46aa655ea8a8efb99dfbbb

hohav / peppi

Switch from arrow2 to arrow-rs since arrow2 has been deprecated #68