search

JuliaDataCubes / YAXArrays.jl

Yet Another XArray-like Julia package
https://juliadatacubes.github.io/YAXArrays.jl/
Other
99 stars 16 forks source link

cannot index Ti after saving to NetCDF #303

Closed bjarthur closed 1 year ago

bjarthur commented 1 year ago

following along in the docs again about how indexing in YAXarrays is fragile so convert to a DimArray first, everything works fine:

julia> using YAXArrays

julia> using DimensionalData: DimensionalData as DD

julia> using DimensionalData

julia> using Dates

julia> axlist = (
           Dim{:time}(Date("2022-01-01"):Day(1):Date("2022-01-30")),
           Dim{:lon}(range(1, 10, length=10)),
           Dim{:lat}(range(1, 5, length=15)),
           Dim{:Variable}(["var1", "var2"])
           )
Dim{:time} Date("2022-01-01"):Day(1):Date("2022-01-30"),
Dim{:lon} 1.0:1.0:10.0,
Dim{:lat} 1.0:0.2857142857142857:5.0,
Dim{:Variable} String["var1", "var2"]

julia> data = rand(30, 10, 15, 2);

julia> ds = YAXArray(axlist, data)
30×10×15×2 YAXArray{Float64,4} with dimensions: 
  Dim{:time} Sampled{Date} Date("2022-01-01"):Day(1):Date("2022-01-30") ForwardOrdered Regular Points,
  Dim{:lon} Sampled{Float64} 1.0:1.0:10.0 ForwardOrdered Regular Points,
  Dim{:lat} Sampled{Float64} 1.0:0.2857142857142857:5.0 ForwardOrdered Regular Points,
  Dim{:Variable} Categorical{String} String["var1", "var2"] ForwardOrdered
Total size: 70.31 KB

julia> using DimensionalData, YAXArrayBase

julia> dim = yaxconvert(DimArray, ds)
30×10×15×2 DimArray{Float64,4} with dimensions: 
  Dim{:time} Sampled{Date} Date("2022-01-01"):Day(1):Date("2022-01-30") ForwardOrdered Regular Points,
  Dim{:lon} Sampled{Float64} 1.0:1.0:10.0 ForwardOrdered Regular Points,
  Dim{:lat} Sampled{Float64} 1.0:0.2857142857142857:5.0 ForwardOrdered Regular Points,
  Dim{:Variable} Categorical{String} String["var1", "var2"] ForwardOrdered
[:, :, 1, 1]
                      1.0        2.0        …  7.0        8.0        9.0        10.0
  2022-01-01  0.940961   0.294824      0.228396   0.17472    0.981411    0.835357
  2022-01-02  0.781415   0.350795      0.280187   0.699315   0.726638    0.759117
  2022-01-03  0.281092   0.0146699     0.712186   0.298064   0.827557    0.686594
  2022-01-04  0.470979   0.254875      0.525043   0.0151862  0.155648    0.953777
  2022-01-05  0.620323   0.183929   …  0.151371   0.59754    0.603862    0.570176
  2022-01-06  0.707482   0.0304242     0.80431    0.368971   0.899546    0.976597
  2022-01-07  0.390528   0.333827      0.612087   0.997762   0.513914    0.231069
  2022-01-08  0.196668   0.462363      0.301448   0.751337   0.242838    0.382593
  2022-01-09  0.615582   0.864773      0.839952   0.151042   0.793742    0.812172
  2022-01-10  0.0955598  0.804951   …  0.715      0.745189   0.984975    0.138555
  2022-01-11  0.892585   0.989434      0.407762   0.139128   0.62286     0.0316323
  2022-01-12  0.86411    0.90847       0.019492   0.412987   0.982372    0.329755
  2022-01-13  0.227525   0.568057      0.680218   0.859744   0.749526    0.283703
  2022-01-14  0.237924   0.637833      0.913615   0.730967   0.0353214   0.0236055
  2022-01-15  0.5781     0.484874   …  0.179348   0.683894   0.359346    0.680334
  2022-01-16  0.982377   0.699114      0.716918   0.030885   0.20293     0.988696
  2022-01-17  0.195431   0.77331       0.536982   0.106134   0.268052    0.390626
  2022-01-18  0.992426   0.568895      0.245542   0.0384513  0.897875    0.328723
  2022-01-19  0.688526   0.258567      0.290371   0.460216   0.982048    0.388009
  2022-01-20  0.502584   0.918318   …  0.0748735  0.753014   0.976429    0.192863
  2022-01-21  0.833324   0.900564      0.351326   0.193641   0.536353    0.579997
  2022-01-22  0.356367   0.0116631     0.764097   0.0362018  0.618749    0.013918
  2022-01-23  0.0117038  0.895351      0.682109   0.480718   0.76915     0.0368658
  2022-01-24  0.42908    0.643252      0.124511   0.841723   0.311599    0.624726
  2022-01-25  0.792917   0.40671    …  0.372279   0.851138   0.778741    0.956629
  2022-01-26  0.514145   0.434614      0.80382    0.444342   0.272141    0.818127
  2022-01-27  0.152628   0.315758      0.282572   0.879835   0.989474    0.988317
  2022-01-28  0.821222   0.602623      0.386316   0.855945   0.68133     0.963644
  2022-01-29  0.231234   0.878954      0.79069    0.0240639  0.979174    0.188462
  2022-01-30  0.183874   0.883172   …  0.678827   0.108555   0.815155    0.852902
[and 29 more slices...]

julia> dim[
           time = DD.Between( Date("2022-01-01"),  Date("2022-01-10")),
           lon=DD.Between(1,2),
           Variable = At("var2")
          ]
10×2×15 DimArray{Float64,3} with dimensions: 
  Dim{:time} Sampled{Date} Date("2022-01-01"):Day(1):Date("2022-01-10") ForwardOrdered Regular Points,
  Dim{:lon} Sampled{Float64} 1.0:1.0:2.0 ForwardOrdered Regular Points,
  Dim{:lat} Sampled{Float64} 1.0:0.2857142857142857:5.0 ForwardOrdered Regular Points
and reference dimensions: 
  Dim{:Variable} Categorical{String} String["var2"] ForwardOrdered
[:, :, 1]
                      1.0        2.0
  2022-01-01  0.104654   0.878094
  2022-01-02  0.689001   0.678357
  2022-01-03  0.478613   0.257777
  2022-01-04  0.369591   0.190209
  2022-01-05  0.0477685  0.136863
  2022-01-06  0.118754   0.949264
  2022-01-07  0.147424   0.309887
  2022-01-08  0.820518   0.0952518
  2022-01-09  0.988964   0.264022
  2022-01-10  0.134161   0.0722799
[and 14 more slices...]

but if i then save the original YAXarray to a NetCDF file, load it back in, and convert again to DimArray, then the time index is changed to Ti and i cannot index into that dimension:

julia> using NetCDF

julia> savecube(ds, "ds.netcdf", driver=:netcdf)
30×10×15×2 YAXArray{Float64,4} with dimensions: 
  Dim{:time} Sampled{Date} Date("2022-01-01"):Day(1):Date("2022-01-30") ForwardOrdered Regular Points,
  Dim{:lon} Sampled{Float64} 1.0:1.0:10.0 ForwardOrdered Regular Points,
  Dim{:lat} Sampled{Float64} 1.0:0.2857142857142857:5.0 ForwardOrdered Regular Points,
  Dim{:Variable} Categorical{String} String["var1", "var2"] ForwardOrdered
Total size: 70.31 KB

julia> DS = Cube("ds.netcdf")
30×10×15×2 YAXArray{Float64,4} with dimensions: 
  Ti Sampled{DateTime} DateTime[2022-01-01T00:00:00, …, 2022-01-30T00:00:00] ForwardOrdered Irregular Points,
  Dim{:lon} Sampled{Float64} 1.0:1.0:10.0 ForwardOrdered Regular Points,
  Dim{:lat} Sampled{Float64} 1.0:0.2857142857142857:5.0 ForwardOrdered Regular Points,
  Dim{:Variable} Categorical{String} String["var1", "var2"] ForwardOrdered
Total size: 70.31 KB

julia> DIM = yaxconvert(DimArray, DS)
30×10×15×2 DimArray{Float64,4} with dimensions: 
  Dim{:Ti} Sampled{DateTime} DateTime[2022-01-01T00:00:00, …, 2022-01-30T00:00:00] ForwardOrdered Irregular Points,
  Dim{:lon} Sampled{Float64} 1.0:1.0:10.0 ForwardOrdered Regular Points,
  Dim{:lat} Sampled{Float64} 1.0:0.2857142857142857:5.0 ForwardOrdered Regular Points,
  Dim{:Variable} Categorical{String} String["var1", "var2"] ForwardOrdered
[:, :, 1, 1]
                                   1.0        …  7.0        8.0        9.0        10.0
  2022-01-01T00:00:00  0.940961      0.228396   0.17472    0.981411    0.835357
  2022-01-02T00:00:00  0.781415      0.280187   0.699315   0.726638    0.759117
  2022-01-03T00:00:00  0.281092      0.712186   0.298064   0.827557    0.686594
  2022-01-04T00:00:00  0.470979      0.525043   0.0151862  0.155648    0.953777
  2022-01-05T00:00:00  0.620323   …  0.151371   0.59754    0.603862    0.570176
  2022-01-06T00:00:00  0.707482      0.80431    0.368971   0.899546    0.976597
  2022-01-07T00:00:00  0.390528      0.612087   0.997762   0.513914    0.231069
  2022-01-08T00:00:00  0.196668      0.301448   0.751337   0.242838    0.382593
  2022-01-09T00:00:00  0.615582      0.839952   0.151042   0.793742    0.812172
  2022-01-10T00:00:00  0.0955598  …  0.715      0.745189   0.984975    0.138555
  2022-01-11T00:00:00  0.892585      0.407762   0.139128   0.62286     0.0316323
  2022-01-12T00:00:00  0.86411       0.019492   0.412987   0.982372    0.329755
  2022-01-13T00:00:00  0.227525      0.680218   0.859744   0.749526    0.283703
  2022-01-14T00:00:00  0.237924      0.913615   0.730967   0.0353214   0.0236055
  2022-01-15T00:00:00  0.5781     …  0.179348   0.683894   0.359346    0.680334
  2022-01-16T00:00:00  0.982377      0.716918   0.030885   0.20293     0.988696
  2022-01-17T00:00:00  0.195431      0.536982   0.106134   0.268052    0.390626
  2022-01-18T00:00:00  0.992426      0.245542   0.0384513  0.897875    0.328723
  2022-01-19T00:00:00  0.688526      0.290371   0.460216   0.982048    0.388009
  2022-01-20T00:00:00  0.502584   …  0.0748735  0.753014   0.976429    0.192863
  2022-01-21T00:00:00  0.833324      0.351326   0.193641   0.536353    0.579997
  2022-01-22T00:00:00  0.356367      0.764097   0.0362018  0.618749    0.013918
  2022-01-23T00:00:00  0.0117038     0.682109   0.480718   0.76915     0.0368658
  2022-01-24T00:00:00  0.42908       0.124511   0.841723   0.311599    0.624726
  2022-01-25T00:00:00  0.792917   …  0.372279   0.851138   0.778741    0.956629
  2022-01-26T00:00:00  0.514145      0.80382    0.444342   0.272141    0.818127
  2022-01-27T00:00:00  0.152628      0.282572   0.879835   0.989474    0.988317
  2022-01-28T00:00:00  0.821222      0.386316   0.855945   0.68133     0.963644
  2022-01-29T00:00:00  0.231234      0.79069    0.0240639  0.979174    0.188462
  2022-01-30T00:00:00  0.183874   …  0.678827   0.108555   0.815155    0.852902
[and 29 more slices...]

julia> DIM[
           Ti = DD.Between( Date("2022-01-01"),  Date("2022-01-10")),
           lon=DD.Between(1,2),
           Variable = At("var2")
          ]
┌ Warning: (Ti,) dims were not found in object
└ @ DimensionalData.Dimensions /groups/scicompsoft/home/arthurb/.julia/packages/DimensionalData/pS9IE/src/Dimensions/primitives.jl:659
30×2×15 DimArray{Float64,3} with dimensions: 
  Dim{:Ti} Sampled{DateTime} DateTime[2022-01-01T00:00:00, …, 2022-01-30T00:00:00] ForwardOrdered Irregular Points,
  Dim{:lon} Sampled{Float64} 1.0:1.0:2.0 ForwardOrdered Regular Points,
  Dim{:lat} Sampled{Float64} 1.0:0.2857142857142857:5.0 ForwardOrdered Regular Points
and reference dimensions: 
  Dim{:Variable} Categorical{String} String["var2"] ForwardOrdered
[:, :, 1]
                                   1.0        2.0
  2022-01-01T00:00:00  0.104654   0.878094
  2022-01-02T00:00:00  0.689001   0.678357
  2022-01-03T00:00:00  0.478613   0.257777
  2022-01-04T00:00:00  0.369591   0.190209
  2022-01-05T00:00:00  0.0477685  0.136863
  2022-01-06T00:00:00  0.118754   0.949264
  2022-01-07T00:00:00  0.147424   0.309887
  2022-01-08T00:00:00  0.820518   0.0952518
  2022-01-09T00:00:00  0.988964   0.264022
  2022-01-10T00:00:00  0.134161   0.0722799
  2022-01-11T00:00:00  0.629499   0.868076
  2022-01-12T00:00:00  0.0221451  0.342178
  2022-01-13T00:00:00  0.970437   0.964414
  2022-01-14T00:00:00  0.111773   0.926125
  2022-01-15T00:00:00  0.495118   0.333009
  2022-01-16T00:00:00  0.445825   0.941374
  2022-01-17T00:00:00  0.652199   0.276319
  2022-01-18T00:00:00  0.127384   0.864917
  2022-01-19T00:00:00  0.0961752  0.181921
  2022-01-20T00:00:00  0.759085   0.229129
  2022-01-21T00:00:00  0.492247   0.23813
  2022-01-22T00:00:00  0.408099   0.0883392
  2022-01-23T00:00:00  0.246613   0.257849
  2022-01-24T00:00:00  0.717849   0.0755125
  2022-01-25T00:00:00  0.835873   0.945049
  2022-01-26T00:00:00  0.295243   0.325043
  2022-01-27T00:00:00  0.77659    0.744565
  2022-01-28T00:00:00  0.240221   0.863562
  2022-01-29T00:00:00  0.0579734  0.419663
  2022-01-30T00:00:00  0.17626    0.07056
[and 14 more slices...]

the output is 30x2x15 which is smaller that 30×10×15×2 but it should be 10x2x15. so it successfully indexed into lon and Variable but not Ti.

hope i'm not doing something stupid again. thanks!

lazarusA commented 1 year ago

just do time = ... as before, instead of Ti = ..., it should work.

bjarthur commented 1 year ago

nope. i tried that earlier, just didn't show it. here it is:

julia> DIM[
           time = DD.Between( Date("2022-01-01"),  Date("2022-01-10")),
           lon=DD.Between(1,2),
           Variable = At("var2")
          ]
┌ Warning: (Dim{:time},) dims were not found in object
└ @ DimensionalData.Dimensions /groups/scicompsoft/home/arthurb/.julia/packages/DimensionalData/pS9IE/src/Dimensions/primitives.jl:659
30×2×15 DimArray{Float64,3} with dimensions: 
  Dim{:Ti} Sampled{DateTime} DateTime[2022-01-01T00:00:00, …, 2022-01-30T00:00:00] ForwardOrdered Irregular Points,
  Dim{:lon} Sampled{Float64} 1.0:1.0:2.0 ForwardOrdered Regular Points,
  Dim{:lat} Sampled{Float64} 1.0:0.2857142857142857:5.0 ForwardOrdered Regular Points
and reference dimensions: 
  Dim{:Variable} Categorical{String} String["var2"] ForwardOrdered
[:, :, 1]
                                   1.0        2.0
  2022-01-01T00:00:00  0.104654   0.878094
  2022-01-02T00:00:00  0.689001   0.678357
  2022-01-03T00:00:00  0.478613   0.257777
  2022-01-04T00:00:00  0.369591   0.190209
  2022-01-05T00:00:00  0.0477685  0.136863
  2022-01-06T00:00:00  0.118754   0.949264
  2022-01-07T00:00:00  0.147424   0.309887
  2022-01-08T00:00:00  0.820518   0.0952518
  2022-01-09T00:00:00  0.988964   0.264022
  2022-01-10T00:00:00  0.134161   0.0722799
  2022-01-11T00:00:00  0.629499   0.868076
  2022-01-12T00:00:00  0.0221451  0.342178
  2022-01-13T00:00:00  0.970437   0.964414
  2022-01-14T00:00:00  0.111773   0.926125
  2022-01-15T00:00:00  0.495118   0.333009
  2022-01-16T00:00:00  0.445825   0.941374
  2022-01-17T00:00:00  0.652199   0.276319
  2022-01-18T00:00:00  0.127384   0.864917
  2022-01-19T00:00:00  0.0961752  0.181921
  2022-01-20T00:00:00  0.759085   0.229129
  2022-01-21T00:00:00  0.492247   0.23813
  2022-01-22T00:00:00  0.408099   0.0883392
  2022-01-23T00:00:00  0.246613   0.257849
  2022-01-24T00:00:00  0.717849   0.0755125
  2022-01-25T00:00:00  0.835873   0.945049
  2022-01-26T00:00:00  0.295243   0.325043
  2022-01-27T00:00:00  0.77659    0.744565
  2022-01-28T00:00:00  0.240221   0.863562
  2022-01-29T00:00:00  0.0579734  0.419663
  2022-01-30T00:00:00  0.17626    0.07056
[and 14 more slices...]
lazarusA commented 1 year ago

it looks like you need to do DateTime(your_date).

bjarthur commented 1 year ago

i'm not sure what you mean. can you please elaborate?

you did notice the warning?

┌ Warning: (Ti,) dims were not found in object
└ @ DimensionalData.Dimensions /groups/scicompsoft/home/arthurb/.julia/packages/DimensionalData/pS9IE/src/Dimensions/primitives.jl:659
bjarthur commented 1 year ago

if changing Date to DateTime in the indexing is what you meant, it doesn't work:

julia> DIM[
           Ti = DD.Between( DateTime("2022-01-01"),  DateTime("2022-01-10")),
           lon=DD.Between(1,2),
           Variable = At("var2")
          ]
┌ Warning: (Ti,) dims were not found in object
└ @ DimensionalData.Dimensions /groups/scicompsoft/home/arthurb/.julia/packages/DimensionalData/pS9IE/src/Dimensions/primitives.jl:659
30×2×15 DimArray{Float64,3} with dimensions: 
  Dim{:Ti} Sampled{DateTime} DateTime[2022-01-01T00:00:00, …, 2022-01-30T00:00:00] ForwardOrdered Irregular Points,
  Dim{:lon} Sampled{Float64} 1.0:1.0:2.0 ForwardOrdered Regular Points,
  Dim{:lat} Sampled{Float64} 1.0:0.2857142857142857:5.0 ForwardOrdered Regular Points
and reference dimensions: 
  Dim{:Variable} Categorical{String} String["var2"] ForwardOrdered
[:, :, 1]
                                   1.0        2.0
  2022-01-01T00:00:00  0.104654   0.878094
  2022-01-02T00:00:00  0.689001   0.678357
  2022-01-03T00:00:00  0.478613   0.257777
  2022-01-04T00:00:00  0.369591   0.190209
  2022-01-05T00:00:00  0.0477685  0.136863
  2022-01-06T00:00:00  0.118754   0.949264
  2022-01-07T00:00:00  0.147424   0.309887
  2022-01-08T00:00:00  0.820518   0.0952518
  2022-01-09T00:00:00  0.988964   0.264022
  2022-01-10T00:00:00  0.134161   0.0722799
  2022-01-11T00:00:00  0.629499   0.868076
  2022-01-12T00:00:00  0.0221451  0.342178
  2022-01-13T00:00:00  0.970437   0.964414
  2022-01-14T00:00:00  0.111773   0.926125
  2022-01-15T00:00:00  0.495118   0.333009
  2022-01-16T00:00:00  0.445825   0.941374
  2022-01-17T00:00:00  0.652199   0.276319
  2022-01-18T00:00:00  0.127384   0.864917
  2022-01-19T00:00:00  0.0961752  0.181921
  2022-01-20T00:00:00  0.759085   0.229129
  2022-01-21T00:00:00  0.492247   0.23813
  2022-01-22T00:00:00  0.408099   0.0883392
  2022-01-23T00:00:00  0.246613   0.257849
  2022-01-24T00:00:00  0.717849   0.0755125
  2022-01-25T00:00:00  0.835873   0.945049
  2022-01-26T00:00:00  0.295243   0.325043
  2022-01-27T00:00:00  0.77659    0.744565
  2022-01-28T00:00:00  0.240221   0.863562
  2022-01-29T00:00:00  0.0579734  0.419663
  2022-01-30T00:00:00  0.17626    0.07056
[and 14 more slices...]
lazarusA commented 1 year ago

ahh, now I see. Ok, your are in the DimensionalData.jl realm. So you need to checkout the docs there. Either way, the issue is that now the full name has changed to Dim{:Ti}, so you need to use that, according to the docs. Hence, your example works with

DIM[
    Dim{:Ti}(DD.Between( Date("2022-01-01"),  Date("2022-01-10"))),
    lon=DD.Between(1,2),
    Variable = At("var2")
    ]

by the way, I updated the docs already with all your previous feedback, hopefully it will be useful for others as well. This Ti, Time, time thing is really annoying at the moment, https://github.com/rafaqz/DimensionalData.jl/issues/499,

bjarthur commented 1 year ago

thanks! i'm sure others will appreciate updated docs too.