search

xcube-dev / xcube

xcube is a Python package for generating and exploiting data cubes powered by xarray, dask, and zarr.
https://xcube.readthedocs.io/
MIT License
199 stars 18 forks source link

xcube gen cannot handle time duplicates properly #139

Closed AliceBalfanz closed 5 years ago

AliceBalfanz commented 5 years ago

Describe the bug When a time stamp is added to an existing data cube a second time, xcube gen either adds the time stamp to the cube, or throws an error stating, that this case cannot be handled yet by xcube gen. In case a time duplicate is added to the data cube without throwing the error, the data cubes' time continuity is destroyed. This case happens due to rounding the time stamp within xcube gen, which can cause that same time stamps are handled as different ones but when they are added to the data cube, the time gets the same.

To Reproduce Steps to reproduce the behavior:

  1. Go to 'fs1/tmp/forGunnar/OLCI_minimal_working_example
  2. Take one or several input files and make sure to duplicate at least one, and the two .yml files containing the cube configuration
  3. Build the data cube: xcube gen --traceback --config l2c-cyanoalert-olci-dan-Romania-2019.yml --config CyanoAlert-config-general.yml --append Z:\temp\forGunnar\OLCI_miminal_working_example\*.nc
  4. Open data cube with xarray and view the times

Expected behavior When passing the same time stamp again to a data cube, it should replace the existing time stamp by the values of the new input file. This could be very useful for updating data in data cubes e.g. with consolidated data.

forman commented 5 years ago

Actually, this is not a bug, but instead expected behavior, because we know that inserting same time slice at existing time isn't implemented yet. Therefore this issue should be an enhancement, which will allow xcube gen to also overwrite existing time slices.

AliceBalfanz commented 5 years ago

But that a time slice is added twice, without giving an error is a bug, isn't it? Because the expected behaviour would be the error.

In case a time duplicate is added to the data cube without throwing the error, the data cubes' time continuity is destroyed. This case happens due to rounding the time stamp within xcube gen, which can cause that same time stamps are handled as different ones but when they are added to the data cube, the time gets the same.

forman commented 5 years ago

Correct.

AliceBalfanz commented 5 years ago

Closing during meeting with @forman