Open bjarthur opened 1 year ago
felixcremer
commented
1 year ago You need to load YAXArrayBase and we should clarify this in the docs. The long term plan is to make the conversion unnecessary at least for DimensioalArrays. You could also try to use the savecube function on the DimArray directly without conversion. Since version 0.5 YAXArrays uses DimensionalData internally.
bjarthur
commented
1 year ago doh. in fact, needing YAXArrayBase is documented on the very page i linked. i just missed it. sorry, and thanks.
okay, so following along those docs from the top of the page, i get an error when showing the output of the conversion:
julia> dim = yaxconvert(DimArray, ds)
30×10×15×2 DimArray{Float64,4} with dimensions:
Dim{:time} Sampled{Date} Date("2022-01-01"):Day(1):Date("2022-01-30") ForwardOrdered Regular Points,
Dim{:lon} Sampled{Float64} 1.0:1.0:10.0 ForwardOrdered Regular Points,
Dim{:lat} Sampled{Float64} 1.0:0.2857142857142857:5.0 ForwardOrdered Regular Points,
Dim{:Variable} Categorical{String} String["var1", "var2"] ForwardOrdered
[:, :, 1, 1]
Error showing value of type DimArray{Float64, 4, Tuple{Dim{:time, DimensionalData.Dimensions.LookupArrays.Sampled{Date, StepRange{Date, Day}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Day}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:lon, DimensionalData.Dimensions.LookupArrays.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:lat, DimensionalData.Dimensions.LookupArrays.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:Variable, DimensionalData.Dimensions.LookupArrays.Categorical{String, Vector{String}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Array{Float64, 4}, DimensionalData.NoName, Dict{String, Any}}:
ERROR: Lookups overlap or are not in order at 2022-01-15 and 2022-01-15and though yaxconverting a subset of a DimArray back to YAXArray as demonstrated in the YAXArray docs works, converting a DimArray that was created as such, as in the docs for DimensionalData, does not:
julia> using DimensionalData
julia> A = rand(X(50), Y(10.0:40.0))
50×31 DimArray{Float64,2} with dimensions:
X,
Y Sampled{Float64} 10.0:1.0:40.0 ForwardOrdered Regular Points
10.0 11.0 12.0 13.0 … 38.0 39.0 40.0
0.551876 0.0671004 0.961211 0.329316 0.00824898 0.638931 0.408412
0.555526 0.542853 0.9268 0.853312 0.761852 0.081645 0.154222
0.6243 0.50352 0.0356045 0.00962141 0.0674333 0.500321 0.655994
0.719308 0.0484075 0.851234 0.0311157 0.957866 0.881494 0.452655
0.171072 0.345615 0.236108 0.502687 … 0.611146 0.783291 0.281872
0.809951 0.387474 0.394327 0.772617 0.274723 0.792827 0.421075
0.518043 0.684303 0.0131663 0.752819 0.725358 0.566616 0.243544
0.580212 0.898939 0.0667302 0.47279 0.533915 0.54374 0.467786
0.578125 0.78626 0.982852 0.942122 0.955552 0.661442 0.961473
0.864014 0.881012 0.144593 0.734155 … 0.59873 0.225211 0.505407
0.507311 0.297512 0.486179 0.664663 0.0782181 0.305352 0.00574907
0.122751 0.398858 0.856911 0.869954 0.340969 0.663311 0.236891
0.614106 0.429924 0.385895 0.615486 0.729074 0.363152 0.679494
0.827123 0.81563 0.908682 0.373007 0.449758 0.155345 0.144505
⋮ ⋱ ⋮
0.317299 0.425185 0.216379 0.887328 0.925633 0.413373 0.109993
0.771927 0.890942 0.273482 0.398071 0.788144 0.189702 0.782103
0.194516 0.0876607 0.37452 0.0966418 0.720949 0.305293 0.423926
0.404393 0.980967 0.87512 0.960275 … 0.789058 0.686704 0.265909
0.677634 0.707456 0.544522 0.820837 0.246203 0.828991 0.302738
0.187551 0.672093 0.695891 0.0722544 0.94657 0.433121 0.956936
0.45732 0.228346 0.347599 0.566268 0.343101 0.602927 0.345002
0.241566 0.443836 0.909689 0.54991 0.422589 0.0469856 0.269347
0.938177 0.879758 0.86941 0.956951 … 0.556482 0.221584 0.625994
0.644881 0.971069 0.0750275 0.720524 0.0288233 0.738464 0.2819
0.359579 0.989729 0.577934 0.0628705 0.731525 0.366281 0.389653
0.201807 0.18214 0.563899 0.689889 0.910816 0.334008 0.350888
0.756964 0.736997 0.383178 0.504977 0.486544 0.460765 0.660528
0.722019 0.851155 0.177384 0.333402 … 0.59969 0.850551 0.776403
julia> using YAXArrays, YAXArrayBase
julia> yaxconvert(YAXArray, A)
ERROR: MethodError: Cannot `convert` an object of type
DimensionalData.Dimensions.LookupArrays.NoMetadata to an object of type
Dict{String}
Closest candidates are:
convert(::Type{T}, ::T) where T<:AbstractDict
@ Base abstractdict.jl:565
convert(::Type{T}, ::AbstractDict) where T<:AbstractDict
@ Base abstractdict.jl:567
convert(::Type{T}, ::T) where T
@ Base Base.jl:64this is with julia 1.9.2 and the master branches of YAXArrays, YAXArrayBase, and DimensionalData. on a mac.
bjarthur
commented
1 year ago i also tried savecube on a DimArray as suggested and it does not work:
julia> using DimensionalData
julia> A = rand(X(50), Y(10.0:40.0))
50×31 DimArray{Float64,2} with dimensions:
X,
Y Sampled{Float64} 10.0:1.0:40.0 ForwardOrdered Regular Points
10.0 11.0 12.0 13.0 … 38.0 39.0 40.0
0.881287 0.629509 0.322425 0.192821 0.735262 0.452089 0.929576
0.71166 0.730386 0.195872 0.545969 0.837671 0.559783 0.746776
0.308719 0.821133 0.23253 0.00672459 0.310229 0.339925 0.981813
0.132109 0.159522 0.835997 0.13891 0.342058 0.658513 0.218166
0.617631 0.555759 0.630413 0.101652 … 0.644106 0.00982902 0.581067
0.222884 0.00914366 0.770757 0.704762 0.0601903 0.667565 0.236102
0.330236 0.6275 0.961671 0.343005 0.0853408 0.660941 0.695706
0.393381 0.461041 0.455061 0.983439 0.347273 0.934276 0.656817
0.321412 0.623917 0.816849 0.679605 0.177762 0.207565 0.806719
0.748949 0.952613 0.951004 0.977449 … 0.680954 0.843969 0.517942
0.470153 0.894079 0.114897 0.672928 0.13251 0.911878 0.938442
0.878412 0.359869 0.543454 0.933525 0.170847 0.262136 0.610139
0.624271 0.409461 0.395193 0.42783 0.733056 0.259033 0.520604
0.889335 0.703164 0.341896 0.867595 0.73462 0.489358 0.249999
⋮ ⋱ ⋮
0.548636 0.674991 0.618104 0.0306931 0.576078 0.921138 0.444401
0.559885 0.979166 0.617086 0.664534 0.0955654 0.655609 0.158219
0.804452 0.0814024 0.560151 0.425425 0.148086 0.999522 0.413779
0.266441 0.382251 0.405739 0.857045 … 0.110575 0.169862 0.669286
0.669948 0.0458019 0.275357 0.305383 0.446119 0.847235 0.307079
0.768782 0.652949 0.179838 0.789262 0.75623 0.00367788 0.662351
0.810297 0.254601 0.330307 0.918308 0.783549 0.985174 0.802206
0.447934 0.850845 0.0587327 0.332171 0.161717 0.149704 0.125113
0.444282 0.771794 0.217707 0.761087 … 0.50669 0.457434 0.777414
0.482268 0.399347 0.463964 0.886698 0.408144 0.966391 0.265343
0.301592 0.357324 0.296951 0.902823 0.84001 0.0614831 0.901588
0.414188 0.699354 0.913981 0.690822 0.271221 0.071632 0.381269
0.548247 0.713637 0.0305305 0.144523 0.58663 0.814366 0.268899
0.161776 0.247457 0.984563 0.424615 … 0.955446 0.639996 0.109052
julia> using YAXArrays, YAXArrayBase, Zarr
julia> savecube(A, "foo.zarr", driver=:zarr)
ERROR: type DimArray has no field properties
Stacktrace:
[1] getproperty
@ ./Base.jl:37 [inlined]
[2] top-level scope
@ REPL[5]:1works fine for me on a YAXArray though:
julia> a = YAXArray(rand(10,20))
10×20 YAXArray{Float64,2} with dimensions:
Dim{:Dim_1} Sampled{Int64} Base.OneTo(10) ForwardOrdered Regular Points,
Dim{:Dim_2} Sampled{Int64} Base.OneTo(20) ForwardOrdered Regular Points
Total size: 1.56 KB
julia> savecube(a, "foo.zarr", driver=:zarr)
10×20 YAXArray{Float64,2} with dimensions:
Dim{:Dim_1} Sampled{Int64} Base.OneTo(10) ForwardOrdered Regular Points,
Dim{:Dim_2} Sampled{Int64} Base.OneTo(20) ForwardOrdered Regular Points
Total size: 1.56 KBI cannot reproduce the show error, this seems to be rather a DimensionalData error. Could you please check, whether also constructing the DimArray directly from the data gives you the error?
julia> ddim = DimArray(data, axlist)I can reproduce the yaxconvert(YAXArray, dim) error. This happens, because the yaxconvert function does not properly handle the NoMetadata indicator from DimensionalData. You can circumvent the problem by providing an empty dictionary as metadata with the correct type, but this is something that we would need to fix in the future.
julia> A = rand(X(50), Y(10.0:40.0), metadata=Dict{String, Any}())I misremembered that with savecube of a DimArray. I thought this would work now after we changed the internals to use DimensionalData but apparently this is not the case. I fixed it in #172 but we closed it in favour of switching to DimensionalData internally. Do you think it would be helpful to be able to use savecube on a DimArray?
bjarthur
commented
1 year ago re. the show error, here is the full transcript of my julia session which demonstrates it, again copied straight from the docs:
julia> using YAXArrays
julia> using DimensionalData: DimensionalData as DD
julia> using DimensionalData
julia> a = YAXArray(rand(10, 20, 5))
10×20×5 YAXArray{Float64,3} with dimensions:
Dim{:Dim_1} Sampled{Int64} Base.OneTo(10) ForwardOrdered Regular Points,
Dim{:Dim_2} Sampled{Int64} Base.OneTo(20) ForwardOrdered Regular Points,
Dim{:Dim_3} Sampled{Int64} Base.OneTo(5) ForwardOrdered Regular Points
Total size: 7.81 KB
julia> using Dates
julia> axlist = (
Dim{:time}(Date("2022-01-01"):Day(1):Date("2022-01-30")),
Dim{:lon}(range(1, 10, length=10)),
Dim{:lat}(range(1, 5, length=15)),
Dim{:Variable}(["var1", "var2"])
)
Dim{:time} Date("2022-01-01"):Day(1):Date("2022-01-30"),
Dim{:lon} 1.0:1.0:10.0,
Dim{:lat} 1.0:0.2857142857142857:5.0,
Dim{:Variable} String["var1", "var2"]
julia> data = rand(30, 10, 15, 2);
julia> ds = YAXArray(axlist, data)
30×10×15×2 YAXArray{Float64,4} with dimensions:
Dim{:time} Sampled{Date} Date("2022-01-01"):Day(1):Date("2022-01-30") ForwardOrdered Regular Points,
Dim{:lon} Sampled{Float64} 1.0:1.0:10.0 ForwardOrdered Regular Points,
Dim{:lat} Sampled{Float64} 1.0:0.2857142857142857:5.0 ForwardOrdered Regular Points,
Dim{:Variable} Categorical{String} String["var1", "var2"] ForwardOrdered
Total size: 70.31 KB
julia> using DimensionalData, YAXArrayBase
julia> dim = yaxconvert(DimArray, ds)
30×10×15×2 DimArray{Float64,4} with dimensions:
Dim{:time} Sampled{Date} Date("2022-01-01"):Day(1):Date("2022-01-30") ForwardOrdered Regular Points,
Dim{:lon} Sampled{Float64} 1.0:1.0:10.0 ForwardOrdered Regular Points,
Dim{:lat} Sampled{Float64} 1.0:0.2857142857142857:5.0 ForwardOrdered Regular Points,
Dim{:Variable} Categorical{String} String["var1", "var2"] ForwardOrdered
[:, :, 1, 1]
Error showing value of type DimArray{Float64, 4, Tuple{Dim{:time, DimensionalData.Dimensions.LookupArrays.Sampled{Date, StepRange{Date, Day}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Day}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:lon, DimensionalData.Dimensions.LookupArrays.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:lat, DimensionalData.Dimensions.LookupArrays.Sampled{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Dim{:Variable, DimensionalData.Dimensions.LookupArrays.Categorical{String, Vector{String}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Array{Float64, 4}, DimensionalData.NoName, Dict{String, Any}}:
ERROR: Lookups overlap or are not in order at 2022-01-15 and 2022-01-15Do you think it would be helpful to be able to use savecube on a DimArray?
not if i can convert to a YAXArray and savecube on that. but i still can't get that to work. thanks for the metadata tip above. that helps. but now the problem seems to be that the elements in one of my Dims are symbols:
julia> using DimensionalData
julia> DimensionalData.@dim MyDim1
julia> A = rand(MyDim1([:x, :y, :z]), metadata = Dict{String, Any}())
3-element DimArray{Float64,1} with dimensions:
MyDim1 Categorical{Symbol} Symbol[:x, :y, :z] ForwardOrdered
:x 0.17538
:y 0.148049
:z 0.11512
julia> using YAXArrays, YAXArrayBase
julia> AY = yaxconvert(YAXArray, A)
3-element YAXArray{Float64,1} with dimensions:
Dim{:MyDim1} Categorical{Symbol} Symbol[:x, :y, :z] ForwardOrdered
Total size: 24.0 bytes
julia> A[1]
0.17538010986958785
julia> AY[1]
0.17538010986958785
julia> using NetCDF
julia> savecube(AY, "AY.netcdf", driver=:netcdf)
ERROR: ArgumentError: array must be non-emptyno problem if the elements are numbers:
julia> B = rand(MyDim1([1, 2, 3]), metadata = Dict{String, Any}())
3-element DimArray{Float64,1} with dimensions:
MyDim1 Sampled{Int64} Int64[1, 2, 3] ForwardOrdered Irregular Points
1 0.340417
2 0.0840058
3 0.646981
julia> BY = yaxconvert(YAXArray, B)
3-element YAXArray{Float64,1} with dimensions:
Dim{:MyDim1} Sampled{Int64} Int64[1, 2, 3] ForwardOrdered Irregular Points
Total size: 24.0 bytes
julia> savecube(BY, "BY.netcdf", driver=:netcdf)
3-element YAXArray{Float64,1} with dimensions:
Dim{:MyDim1} Sampled{Int64} Int64[1, 2, 3] ForwardOrdered Irregular Points
Total size: 24.0 bytesI copied your show example into the REPL in a new environment with the latest YAXArrays, YAXArrayBase, DimensionalData versions and it works for me and it looks like this:
```julia julia> dim = yaxconvert(DimArray, ds) 30×10×15×2 DimArray{Float64,4} with dimensions: Dim{:time} Sampled{Date} Date("2022-01-01"):Day(1):Date("2022-01-30") ForwardOrdered Regular Points, Dim{:lon} Sampled{Float64} 1.0:1.0:10.0 ForwardOrdered Regular Points, Dim{:lat} Sampled{Float64} 1.0:0.2857142857142857:5.0 ForwardOrdered Regular Points, Dim{:Variable} Categorical{String} String["var1", "var2"] ForwardOrdered [:, :, 1, 1] 1.0 2.0 … 7.0 8.0 9.0 10.0 2022-01-01 0.25224 0.57937 0.506676 0.00119183 0.455278 0.207376 2022-01-02 0.0557954 0.956016 0.351715 0.839776 0.863212 0.00934017 2022-01-03 0.730516 0.111143 0.217946 0.683535 0.662452 0.702881 2022-01-04 0.831242 0.555042 0.0806665 0.55273 0.759573 0.802987 2022-01-05 0.106859 0.402748 … 0.478955 0.148713 0.939527 0.254214 2022-01-06 0.611891 0.435965 0.408148 0.163618 0.462386 0.266909 2022-01-07 0.778541 0.124514 0.660918 0.82015 0.0510813 0.133885 2022-01-08 0.0864876 0.590058 0.676214 0.593179 0.829276 0.504266 2022-01-09 0.611419 0.708731 0.085667 0.544805 0.512177 0.533554 2022-01-10 0.268984 0.641603 … 0.263332 0.387504 0.0745554 0.0355572 2022-01-11 0.135782 0.808536 0.159677 0.652452 0.427886 0.685817 2022-01-12 0.213845 0.115216 0.158461 0.8521 0.719588 0.762179 2022-01-13 0.308991 0.809956 0.127138 0.742123 0.438436 0.170422 2022-01-14 0.729415 0.16056 0.708445 0.728159 0.21577 0.738906 2022-01-15 0.769786 0.321205 … 0.100973 0.278482 0.23814 0.14518 2022-01-16 0.0493651 0.0557759 0.0976277 0.653383 0.0107009 0.848799 2022-01-17 0.73619 0.364057 0.758723 0.797984 0.651297 0.0627265 2022-01-18 0.77829 0.342004 0.236923 0.803619 0.323583 0.569309 2022-01-19 0.896427 0.734217 0.96433 0.146138 0.733856 0.295015 2022-01-20 0.147742 0.617593 … 0.235058 0.19522 0.720362 0.444467 2022-01-21 0.508874 0.168179 0.118952 0.504789 0.679678 0.50959 2022-01-22 0.315509 0.524227 0.527727 0.294725 0.75189 0.204255 2022-01-23 0.277914 0.533151 0.215969 0.278824 0.0990664 0.192069 2022-01-24 0.935064 0.595783 0.46333 0.0741327 0.131488 0.265558 2022-01-25 0.326445 0.910082 … 0.309823 0.946742 0.102515 0.0188348 2022-01-26 0.0198457 0.434458 0.866245 0.671358 0.954846 0.706557 2022-01-27 0.0086935 0.437067 0.23929 0.915465 0.872483 0.252025 2022-01-28 0.646609 0.0948915 0.157426 0.172049 0.229253 0.165325 2022-01-29 0.783618 0.0249393 0.667314 0.6306 0.100676 0.842124 2022-01-30 0.27857 0.240208 … 0.816217 0.987482 0.772402 0.190181 [and 29 more slices...] ```
The main difference in the show method between DimensionalData and YAXArrays is, that YAXArrays doesn't try to show the elements of the array because we mainly deal with larger than memory data and it is quite costly to even show a few values at the edges.
Could you please check whether using only DimensionalData gives you the same error? This might be a minimal example for that:
julia> using DimensionalData
julia> axlist = (
Dim{:time}(Date("2022-01-01"):Day(1):Date("2022-01-30")),
Dim{:lon}(range(1, 10, length=10)),
Dim{:lat}(range(1, 5, length=15)),
Dim{:Variable}(["var1", "var2"])
)
Dim{:time} Date("2022-01-01"):Day(1):Date("2022-01-30"),
Dim{:lon} 1.0:1.0:10.0,
Dim{:lat} 1.0:0.2857142857142857:5.0,
Dim{:Variable} String["var1", "var2"]
julia> data = rand(30, 10, 15, 2);
julia> ddim = DimArray(data, axlist)Regarding the saving of a YAXArray with Symbol entries in the Dimension. It seems to be not possible to save the Symbols directly into netcdf, because also Rasters runs into similar issues. As a quick fix you can convert the symbols into strings and then save it.
felixcremer
commented
1 year ago We could convert the symbols to strings in the background to hide this step from the user, but then we might have to save somewhere in the metadata that this dimension is supposed to be symbols because otherwise we would get a different YAXArray by saving to disk and reopening the dataset and this might be confusing.
bjarthur
commented
1 year ago mysteriously, i can not repeat the show error now. even in same environment. perhaps i upgraded it and don't remember. anyway, thanks!
yaxconvertis mentioned in the docs, and three issues (#169, #194, #229), but i don't see a definition anywhere in the code in the entirety of the git history (pkg.dev, git log -p, then search). what am i missing?i have a DimensionalData array, and would like to convert it to a YAXArray so that i can save it to a chunked file and then memory map just the parts i need. that's what yaxconvert is documented to do IIUC. is there another way?
thanks!