jorgecarleitao / arrow2

Transmute-free Rust library to work with the Arrow format
Apache License 2.0
1.06k stars 222 forks source link

MutableListArray from an existing MutableArray #1502

Open cjermain opened 1 year ago

cjermain commented 1 year ago

Is there a way to directly nest a non-empty MutableArray in a MutableListArray?

In the majority of MutableArray implementations there is a from_trusted_len_iter method that allows the array to be constructed directly from an iterator. In the case of a MutableStructArray, the new method takes a vector of the inner MutableArrays, so you can directly construct it. Is there an analogy for MutableListArray?

It looks like the new_from has an assert that prevents non-empty MutableArrays from being passed through: https://github.com/jorgecarleitao/arrow2/blob/fb5e4d591c7149df590a330365fae55d2370962f/src/array/list/mutable.rs#L118

Along with the MutableArray I also have all of the lengths in a Vec<Option<usize>>. I had been hoping to use the following, but am failing on the assert during runtime:

let mut array = MutableListArray::<i32, _>::new_with_capacity(inner_array, lengths.len());
array.try_extend_from_lengths(lengths).unwrap();

I'm trying to accomplish the same thing in this function, without the pre-allocation of an empty inner MutableArray ahead of time: https://github.com/jorgecarleitao/arrow2/blob/fb5e4d591c7149df590a330365fae55d2370962f/src/io/json/read/deserialize.rs#L254-L277

cjermain commented 1 year ago

@jorgecarleitao, what do you think?