Open nalimilan opened 7 years ago
Why not just do this in DataFrames/DataArrays, since the approach there is already the closest to how Nulls.jl works?
Because storing columns as Array{Union{T, Null}}
is going to be quite slow until (at least) Julia 1.0, and because AFAIK we don't want to continue using Nullable
in the future. So better keep DataFrames usable for now (maybe porting DataArrays to Null
, but keeping them for efficient memory layout) and apply breaking changes to DataTables, which are still in an experimental state. After Julia 1.0 we should be able to make DataFrames and DataTables converge to a common representation.
I would much prefer if we could keep DataTables.jl as the place for a container based approach to missing values. At this point it is not clear whether the Union{T,Null}
approach can work for the whole data ecosystem (e.g. Query.jl) and I don't think we should start to convert anything in this repo here until that is sorted out.
Why not do the Nulls.jl work in a branch in DataFrames.jl?
@quinnj has just created a Null.jl package providing a new
Null
type to replace DataArrays'NAtype
. Even if the Julia compiler doesn't yet include the necessary optimizations to handleUnion{T, Null}
efficiently (see e.g. discussion at https://github.com/JuliaData/Nulls.jl/issues/3), I think we should start moving away fromNullable
now, so that at least we can stabilize the API even if performance remains poor for some time.NullableArray
can be replaced withArray{Union{T, Null}}
, which Jameson said will eventually use the same memory layout asNullableArray
. This should suit quite well with @cjprybol's PR https://github.com/JuliaData/DataTables.jl/pull/53 which is going to stop auto-promoting columns toNullableArray
.CategoricalArray
andNullableCategoricalArray
will have to be adapted, but that shouldn't be too hard.