Open alamb opened 6 months ago
Hi @alamb
I'm interested in the progress of this issue regarding the addition of ListViewArray and LargeListViewArray to arrow-rs. I've been following the discussion and would like to know if there have been any updates or if there's anything specific I can assist with to help move this forward.
Hi @alamb
I'm interested in the progress of this issue regarding the addition of ListViewArray and LargeListViewArray to arrow-rs. I've been following the discussion and would like to know if there have been any updates or if there's anything specific I can assist with to help move this forward.
Hi @Kikkon -- that is great news. 🙏
There have been discussions related to implementing StringView
and BinaryView
as part of https://github.com/apache/arrow-rs/issues/5374 and I expect that work to begin shortly (within a month maybe?).
I don't know of any similar work afoot for ListViewArray
but I suspect we could follow a very similar pattern to https://github.com/apache/arrow-rs/issues/5374 (e.g. implement the basic array structure first, followed by cast
, take
, and filter
kernels)
Does that make sense?
Hi @alamb I have already created some preliminary issues: https://github.com/apache/arrow-rs/issues/5492 and pull requests https://github.com/apache/arrow-rs/pull/5493 . Regarding the subsequent tasks, if time permits, I can also try it.
Thank you @Kikkon -- that sounds awesome
@alamb Since #5492 has already been merged, I've created an issue #5501 to keep track of the progress. Is there anything that needs to be added?"
@alamb Since #5492 has already been merged, I've created an issue #5501 to keep track of the progress. Is there anything that needs to be added?"
Thanks @Kikkon
I added #5501 to the top of this ticket. In terms of next steps, I think the list of items we are finding on https://github.com/apache/arrow-rs/issues/5374 is worth looking at (e.g. IPC, support for filter/take, etc). We'll keep that list updated
Is your feature request related to a problem or challenge? Please describe what you are trying to do. Recently two new types were added to the Arrow format that make it more suitable for certain types of operations on Lists
Specifically when doing filtering / take with List data, creating a new
ListArray
orLargeListArray
requires copying the underlying lists to a new, packed buffer. The "ListView" was designed to solve this limitation and recently added to the Arrow spec.Describe the solution you'd like I would like to implement
ListViewArray
andLargeListViewArray
following the spec: The spec: https://arrow.apache.org/docs/format/Columnar.html#listview-layoutInitially, I would suggest we get the basic types in place:
DataType
Array
implementations and layout and basic constructionThen as follow on PRs, add support to key kernels:
cast
(to/from ListAray / LargeListArray)filter
take
Describe alternatives you've considered
Additional context This is similar in spirit to the
StringViewArray
andBinaryViewArray
described in https://github.com/apache/arrow-rs/issues/5374Tasks: