Closed felixcremer closed 1 year ago
The currently failing tests are how to access the dimension of a cube by name cube.axname
does not work with DimensionalData and we would need to decide whether we would like to keep it or disallow it.
Then we do not have a refdims method and I am not sure, how we should implement it, because I am not sure, how it is used in DimensionalData.
Another failure is, that the YAXArray now does not throw an exception when there is a length mismatch between the dimensions and the data.
could you pin point exactly where in the code is this happening? Maybe @rafaqz could see a possible solution very quickly, if he happens to have a minute 😄 .
The refdims is more a conceptual question of what do you try to achieve with it. I will have a look into the not throwing with the length mismatch tomorrow.
What would be the way to access a dimension of a DimArray based on the dimension name?
refdims
show where the slice comes from in plot labels, and lets cat
rebuild sliced dimensions. Otherwise it doesnt do much.
Its just a tuple like dims. You can define it to always return an empty tuple if you dont want it.
One gotcha may be the need to run format
on the Dimensions to detect the right LookupArray
properties.
See here: https://github.com/rafaqz/DimensionalData.jl/blob/main/src/array/array.jl#L327
Then if you use strings it will choose a Categorical
lookup, and it will check the array/range ordering so it can use searchsorted
for fast lookups.
Without format
Dimension
s are relatively dumb wrappers. We cant do format
on construction either because dimensions are also used to wrap lots of other things.
@meggart I just pushed the current state of this PR and it "only" fails because of the getindex of the Dataset and I also run into a getindex ambiguity between YAXArray and AbstractDimArray getindex functions. I solved it first by removing the YAXArray getindex method, but then the getindex seems to be not lazy anymore in some cases.
The subsetting is now working. This still fails for a a shift of the look chunks by one and because the call create_dataset for the MockDataset type fails.
I don't understand both tests not well enough to proceed from here.
It seems to be that prependrange is broken for a range of Dates with a nonzero chunkoffset. At least that is the error that I am currently running into.
InexactError: Int64(674238.4615384616)
Stacktrace:
[1] Int64
@ ./float.jl:900 [inlined]
[2] convert
@ ./number.jl:7 [inlined]
[3] Day
@ ~/.julia/juliaup/julia-1.9.0+0.x64.linux.gnu/share/julia/stdlib/v1.9/Dates/src/types.jl:55 [inlined]
[4] *
@ ~/.julia/juliaup/julia-1.9.0+0.x64.linux.gnu/share/julia/stdlib/v1.9/Dates/src/periods.jl:90 [inlined]
[5] *
@ ~/.julia/juliaup/julia-1.9.0+0.x64.linux.gnu/share/julia/stdlib/v1.9/Dates/src/periods.jl:91 [inlined]
[6] lerpi
@ ./range.jl:966 [inlined]
[7] unsafe_getindex(r::LinRange{Day, Int64}, i::Int64)
@ Base ./range.jl:959
[8] getindex
@ ./range.jl:941 [inlined]
[9] copyto_unaliased!(deststyle::IndexLinear, dest::Vector{Day}, srcstyle::IndexLinear, src::LinRange{Day, Int64})
@ Base ./abstractarray.jl:1091
[10] copyto!
@ ./abstractarray.jl:1071 [inlined]
[11] copyto!
@ ./broadcast.jl:967 [inlined]
[12] copyto!
@ ./broadcast.jl:926 [inlined]
[13] materialize!
@ ./broadcast.jl:884 [inlined]
[14] materialize!(dest::Vector{Day}, bc::Base.Broadcast.Broadcasted{Base.Broadcast.DefaultArrayStyle{1}, Nothing, typeof(identity), Tuple{LinRange{Day, Int64}}})
@ Base.Broadcast ./broadcast.jl:881
[15] add_var(ds::MockDataset, x::LinRange{Day, Int64}, name::String, dimlist::Tuple{String}, atts::Dict{String, Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ YAXArrayBase ~/.julia/dev/YAXArrayBase/src/datasets/datasetinterface.jl:50
[16] add_var
@ ~/.julia/dev/YAXArrayBase/src/datasets/datasetinterface.jl:47 [inlined]
[17] create_dataset(T::Type, path::String, gatts::Dict{String, Any}, dimnames::Tuple{String, String, String}, dimvals::Tuple{LinRange{Day, Int64}, Vector{String}, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, dimattrs::Tuple{Dict{String, Any}, Dict{String, Any}, Dict{String, Any}}, vartypes::Vector{DataType}, varnames::Vector{String}, vardims::Vector{Tuple{Symbol, Symbol, Symbol}}, varattrs::Vector{Dict{String, Any}}, varchunks::Vector{Tuple{Int64, Int64, Int64}}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ YAXArrayBase ~/.julia/dev/YAXArrayBase/src/datasets/datasetinterface.jl:59
[18] create_dataset(T::Type, path::String, gatts::Dict{String, Any}, dimnames::Tuple{String, String, String}, dimvals::Tuple{LinRange{Day, Int64}, Vector{String}, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}}, dimattrs::Tuple{Dict{String, Any}, Dict{String, Any}, Dict{String, Any}}, vartypes::Vector{DataType}, varnames::Vector{String}, vardims::Vector{Tuple{Symbol, Symbol, Symbol}}, varattrs::Vector{Dict{String, Any}}, varchunks::Vector{Tuple{Int64, Int64, Int64}})
@ YAXArrayBase ~/.julia/dev/YAXArrayBase/src/datasets/datasetinterface.jl:54
[19] createdataset(DS::Type, axlist::Tuple{DimensionalData.Dimensions.Ti{StepRange{Date, Month}}, DimensionalData.Dimensions.Dim{:Variable, Vector{String}}, DimensionalData.Dimensions.Dim{:Xvals, UnitRange{Int64}}}; path::String, persist::Bool, T::Type, chunksize::Tuple{Int64, Int64, Int64}, chunkoffset::Tuple{Int64, Int64, Int64}, overwrite::Bool, properties::Dict{String, Int64}, datasetaxis::String, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
@ YAXArrays.Datasets ~/.julia/dev/YAXArrays/src/DatasetAPI/Datasets.jl:663
[20] macro expansion
@ ~/.julia/dev/YAXArrays/test/Datasets/datasets.jl:189 [inlined]
[21] macro expansion
@ ~/.julia/juliaup/julia-1.9.0+0.x64.linux.gnu/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined]
[22] macro expansion
@ ~/.julia/dev/YAXArrays/test/Datasets/datasets.jl:175 [inlined]
[23] macro expansion
@ ~/.julia/juliaup/julia-1.9.0+0.x64.linux.gnu/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined]
[24] macro expansion
@ ~/.julia/dev/YAXArrays/test/Datasets/datasets.jl:69 [inlined]
[25] macro expansion
@ ~/.julia/juliaup/julia-1.9.0+0.x64.linux.gnu/share/julia/stdlib/v1.9/Test/src/Test.jl:1498 [inlined]
[26] top-level scope
@ ~/.julia/dev/YAXArrays/test/Datasets/datasets.jl:6
[27] include(fname::String)
@ Base.MainInclude ./client.jl:478
[28] top-level scope
@ ~/.julia/dev/YAXArrays/test/runtests.jl:15
[29] include(fname::String)
@ Base.MainInclude ./client.jl:478
[30] top-level scope
@ none:6
[31] eval
@ ./boot.jl:370 [inlined]
[32] exec_options(opts::Base.JLOptions)
@ Base ./client.jl:280
[33] _start()
@ Base ./client.jl:522
And this happens, because the prependrange function tries to construct a linrange from a StepRange with an offset. I am not sure, why we are now running into this with the change to DimensionalData.
@gdkrmr This branch is now ready and I am going to merge it on thursday as version 0.5 if you don't object.
Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | ||
---|---|---|---|---|---|
src/Cubes/Slices.jl | 0 | 1 | 0.0% | ||
src/DAT/registration.jl | 2 | 3 | 66.67% | ||
src/DAT/DAT.jl | 7 | 9 | 77.78% | ||
src/DAT/tablestats.jl | 4 | 8 | 50.0% | ||
src/Cubes/Cubes.jl | 21 | 29 | 72.41% | ||
src/helpers.jl | 38 | 51 | 74.51% | ||
src/DatasetAPI/Datasets.jl | 16 | 38 | 42.11% | ||
<!-- | Total: | 106 | 157 | 67.52% | --> |
Files with Coverage Reduction | New Missed Lines | % | ||
---|---|---|---|---|
src/Cubes/Rechunker.jl | 2 | 83.82% | ||
src/Cubes/Cubes.jl | 87 | 44.88% | ||
src/DatasetAPI/Datasets.jl | 167 | 34.53% | ||
<!-- | Total: | 256 | --> |
Totals | |
---|---|
Change from base Build 5455825663: | -17.3% |
Covered Lines: | 981 |
Relevant Lines: | 1678 |
Patch coverage: 67.92
% and project coverage change: -17.23
:warning:
Comparison is base (
28fa646
) 75.82% compared to head (ac0eae6
) 58.60%.
:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.
We should be thinking about what to reexport from DimensionalData. Should it just be everything? @rafaqz @meggart
I am currently working through the docs to change to the DD syntax.
DD has limited its exports a lot so it might be ok to just reexport everything now.
The main pain point is there are a few annoying clashes with DataFrames.jl, but im in the process of getting rid of some of them.
The documentation builds locally, but I have no idea how to debug the Documenter build on the Github Actions. I am for now excluded the ESDL examples, because we didn't make the switch with EarthDataLab yet and I also exclude the generate via a function example.
@lazarusA do you have some hints on fixing the Documenter build on CI?
just removing esdl_study1 should do. And then remove pyplot
and PyCall
from the Project.toml
should fix the build in CI, I had read before related issues to conda /pycall in CI is kinda brittle nowadays.
This pull request is going to make the YAXArray type a subtype of AbstractDimArray. With that we will have a better interoperability between YAXArrays.jl and Rasters.jl
The tests are working, but the documentation is not deploying yet so I will fix the documentation and then I am going to merge it.
This PR is heavily breaking, because it changes the indexing syntax.