apache / arrow-julia

Official Julia implementation of Apache Arrow
https://arrow.apache.org/julia/
Other
283 stars 60 forks source link

improved support for struct columns with missing values #498

Open baumgold opened 5 months ago

baumgold commented 5 months ago

The current implementation depends on Arrow.default returning a sentinel object when the value is missing. This usually works, but occasionally has problems. This PR simplifies the implementation and solves some of the problematic corner-cases.

One example of a problematic corner-case is a column of lists of structs:

julia> [(a=1,b=2), missing, (a=3,b=4)]
3-element Vector{Union{Missing, @NamedTuple{a::Int64, b::Int64}}}:
 (a = 1, b = 2)
 missing
 (a = 3, b = 4)

It's possible that users may wrap this Vector in a SubArray. In this case Arrow.default(::Type{<:SubArray }) actually uses Arrow.default(::Type{<:AbstractVector}) which enforces that the parent-type is a Vector - this is not always the case. For example, the parent may well be of type Arrow.Struct if the data being written was also read from an Arrow file. Certainly we could enhance Arrow.default to return a proper type for SubArray but simpler is to remove the dependency on Arrow.default from ToStruct.

codecov-commenter commented 5 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (ac199b0) 87.34% compared to head (78e22be) 87.36%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #498 +/- ## ========================================== + Coverage 87.34% 87.36% +0.01% ========================================== Files 26 26 Lines 3288 3292 +4 ========================================== + Hits 2872 2876 +4 Misses 416 416 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

baumgold commented 4 months ago

@quinnj / @ericphanson - Any questions/comments/concerns here? If not I'd like to merge and release. Thanks.

ericphanson commented 4 months ago

I don't really know this code well enough to be confident here. But one thing I can say is it's probably good if the PR adds a test that is failing on main, to show precisely what has been fixed (and prevent regressions)