Scalar::List does not encapsulate all information from ListArray

[jorgecarleitao]

IMO the roundtrip ListArray -> Scalar::List -> ListArray is currently lossy. This happens because the Scalar::List does not encapsulate everything from the ListArray.

Examples:

• it is not possible to recover the field name and nullability of the inner type.
• nested types are hard to rebuild

(new) suggested signature:

Scalar::List(Option<Arc<dyn Array>>, DataType)

With this, the second argument stores the original datatype, which allows to recover all the information from ListArray, and the first argument is easy to recover and build in both directions:

• ListArray -> Scalar: use array.value(index)
• Scalar -> ListArray: use concatenate(repeat(inner).take(size))

[jorgecarleitao]

@andygrove , I was trying to address this in ballista, but I am struggling to encapsulate a ScalarValue that depends on an Array; how are we declaring arrow arrays in the protobuf? I can't find any reference to them.