Document ChunkedArray Abstractions

[tustvold]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

The question has come up a couple of times as to why we don't have a ChunkedArray abstraction, we should document why we don't and what the equivalent constructions are.

Describe the solution you'd like

A ChunkedArray is really just sugar over the top of Vec<ArrayRef>, and is used within arrow-cpp and pyarrow for representing large in-memory datasets.

Equivalent constructions in arrow-rs would be:

• Vec<ArrayRef>: a fairly exact mirror to ChunkedArray sans some ergonomic niceties like working natively in kernels
• impl Iterator<Item=ArrayRef> a lazy version of a ChunkedArray
• impl Stream<Item=ArrayRef> a lazy async version of a ChunkedArray - this is what DataFusion uses extensively

There are also equivalent constructions using RecordBatch instead of ArrayRef.

These abstractions are strictly more flexible the pyarrow ChunkedArray concept, integrate better with the Rust ecosystem, and encourage users towards lazy evaluation, which has much better memory usage characteristics.

Describe alternatives you've considered

Additional context

[alamb]

impl Stream a lazy async version of a ChunkedArray - this is what DataFusion uses extensively

In case anyone wants details, this is called RecordBatchStream:

https://docs.rs/datafusion/latest/datafusion/execution/trait.RecordBatchStream.html