search

xcube-dev / xcube

xcube is a Python package for generating and exploiting data cubes powered by xarray, dask, and zarr.
https://xcube.readthedocs.io/
MIT License
201 stars 20 forks source link

Unpin pinned dependency versions #939

Closed pont-us closed 8 months ago

pont-us commented 8 months ago

Describe the bug

Currently, the following dependencies are pinned in environment.yml:

In the case of xarray, 61 unit tests fail if the current latest version (2024.2.0) is installed. Of course these pins can't be retained indefinitely, and both pinned packages are already significantly behind the current version. We should fix the incompatibilities and unpin them as soon as possible.

pont-us commented 8 months ago

The initial xarray pin was motivated by #897 ; with the latest (2024.2.0) xarray version, it has been joined by another 60 (!) unit test failures and errors. Fortunately many of them seem to have similar causes, so we probably won't need 61 separate fixes.

pont-us commented 8 months ago

Summary of failing tests with 2024.2.0:

FAILED test/cli/test_dump.py::DumpTest::test_dump_ds - AssertionError: '<xarray.Dataset> Size: 8MB\nDimensions:        (lon: 360, ...
FAILED test/cli/test_level.py::LevelDataTest::test_all_defaults - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/cli/test_level.py::LevelDataTest::test_invalid_inputs - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/cli/test_level.py::LevelDataTest::test_level_with_nc - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/cli/test_level.py::LevelDataTest::test_level_with_zarr - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/cli/test_level.py::LevelDataTest::test_level_with_zarr_agg_method - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/cli/test_level.py::LevelDataTest::test_level_with_zarr_agg_methods - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/cli/test_level.py::LevelDataTest::test_level_with_zarr_link - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/cli/test_level.py::LevelDataTest::test_level_with_zarr_num_levels_max - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/cli/test_level.py::LevelDataTest::test_with_existing_output - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/cli/test_level.py::LevelDataTest::test_with_output - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/cli/test_level.py::LevelDataTest::test_with_tile_size_and_num_levels - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/cli/test_optimize.py::OptimizeDataTest::test_defaults - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/cli/test_optimize.py::OptimizeDataTest::test_in_place - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/cli/test_optimize.py::OptimizeDataTest::test_user_output - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/cli/test_prune.py::PruneDataTest::test_delete_block_file - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/cli/test_prune.py::PruneDataTest::test_dry_run - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/cli/test_prune.py::PruneDataTest::test_no_dry_run - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/mldataset/test_fs.py::FsMultiLevelDatasetTest::test_io_nl_4_ts_256 - xcube.core.mldataset.fs.FsMultiLevelDatasetError: Failed to write dataset t...
FAILED test/core/mldataset/test_fs.py::FsMultiLevelDatasetTest::test_io_nl_4_ts_256_agg - xcube.core.mldataset.fs.FsMultiLevelDatasetError: Failed to write dataset t...
FAILED test/core/store/fs/test_registry.py::FileFsDataStoresTest::test_dataset_levels - xcube.core.store.error.DataStoreError: Failed to write dataset /tmp/xcubeiv...
FAILED test/core/store/fs/test_registry.py::FileFsDataStoresTest::test_dataset_netcdf - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/store/fs/test_registry.py::FileFsDataStoresTest::test_dataset_zarr - xcube.core.store.error.DataStoreError: Failed to write dataset '/tmp/xcubew...
FAILED test/core/store/fs/test_registry.py::FileFsDataStoresTest::test_mldataset_levels - xcube.core.store.error.DataStoreError: Failed to write dataset /tmp/xcubeu0...
FAILED test/core/store/fs/test_registry.py::MemoryFsDataStoresTest::test_dataset_levels - xcube.core.store.error.DataStoreError: Failed to write dataset xcube/testin...
FAILED test/core/store/fs/test_registry.py::MemoryFsDataStoresTest::test_dataset_netcdf - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/store/fs/test_registry.py::MemoryFsDataStoresTest::test_dataset_zarr - xcube.core.store.error.DataStoreError: Failed to write dataset 'xcube/testi...
FAILED test/core/store/fs/test_registry.py::MemoryFsDataStoresTest::test_mldataset_levels - xcube.core.store.error.DataStoreError: Failed to write dataset xcube/testin...
FAILED test/core/store/fs/test_registry.py::S3FsDataStoresTest::test_dataset_levels - xcube.core.store.error.DataStoreError: Failed to write dataset xcube/testin...
FAILED test/core/store/fs/test_registry.py::S3FsDataStoresTest::test_dataset_netcdf - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/store/fs/test_registry.py::S3FsDataStoresTest::test_dataset_zarr - xcube.core.store.error.DataStoreError: Failed to write dataset 'xcube/testi...
FAILED test/core/store/fs/test_registry.py::S3FsDataStoresTest::test_mldataset_levels - xcube.core.store.error.DataStoreError: Failed to write dataset xcube/testin...
FAILED test/core/test_dump.py::DumpDatasetTest::test_dump_dataset - AssertionError: "<xarray.DataArray 'precipitation' (time: 5, lat: 180, lon:...
FAILED test/core/test_normalize.py::TestNormalize::test_normalize_zonal_lat_lon - ValueError: cannot re-index or align objects with conflicting indexes found...
FAILED test/core/test_optimize.py::OptimizeDatasetTest::test_failures - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_optimize.py::OptimizeDatasetTest::test_optimize_dataset - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_optimize.py::OptimizeDatasetTest::test_optimize_dataset_in_place - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_optimize.py::OptimizeDatasetTest::test_optimize_dataset_unchunk_coords_str - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_optimize.py::OptimizeDatasetTest::test_optimize_dataset_unchunk_coords_str_in_place - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_optimize.py::OptimizeDatasetTest::test_optimize_dataset_unchunk_coords_str_list - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_optimize.py::OptimizeDatasetTest::test_optimize_dataset_unchunk_coords_str_list_in_place - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_optimize.py::OptimizeDatasetTest::test_optimize_dataset_unchunk_coords_str_tuple - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_optimize.py::OptimizeDatasetTest::test_optimize_dataset_unchunk_coords_str_tuple_in_place - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_optimize.py::OptimizeDatasetTest::test_optimize_dataset_unchunk_coords_true - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_optimize.py::OptimizeDatasetTest::test_optimize_dataset_unchunk_coords_true_in_place - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_timeslice.py::TimeSliceTest::test_append_time_slice - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_timeslice.py::TimeSliceTest::test_find_time_slice - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_timeslice.py::TimeSliceTest::test_insert_time_slice - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_timeslice.py::TimeSliceTest::test_replace_time_slice - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_timeslice.py::TimeSliceTest::test_update_corrupt_cube - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_timeslice.py::ZarrStoreTest::test_local - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_unchunk.py::UnchunkDatasetTest::test_chunked - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_unchunk.py::UnchunkDatasetTest::test_dont_unchunk_if_unchunked - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_unchunk.py::UnchunkDatasetTest::test_unchunk_all - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_unchunk.py::UnchunkDatasetTest::test_unchunk_coord_var - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_unchunk.py::UnchunkDatasetTest::test_unchunk_coords_only - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_unchunk.py::UnchunkDatasetTest::test_unchunk_data_var - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_unchunk.py::UnchunkDatasetTest::test_unchunk_data_var_coords_only - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_unchunk.py::UnchunkDatasetTest::test_unchunk_invalid_path - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/test_unchunk.py::UnchunkDatasetTest::test_unchunk_invalid_vars - ValueError: When encoding chunked arrays of datetime values, both the units...
FAILED test/core/zarrstore/test_generic.py::CommonS3ZarrStoreTest::test_it - ValueError: When encoding chunked arrays of datetime values, both the units...
ERROR test/core/store/fs/test_registry.py::NewCubeDataTestMixin::test_open_packed - ValueError: When encoding chunked arrays of datetime values, both the units...
ERROR test/core/store/fs/test_registry.py::NewCubeDataTestMixin::test_open_unpacked - ValueError: When encoding chunked arrays of datetime values, both the units...
= 61 failed, 1356 passed, 39 skipped, 6357 warnings, 2 errors in 212.92s (0:03:32) =
pont-us commented 7 months ago

Just for reference, I'm attaching a script I wrote to investigate one of the errors covered by this issue. The script demonstrates the changed xcube behaviour when serializing a chunked datetime array for which the encoding specifies the units but not the (encoded) dtype.

trigger_xarray_encoding_error.py.zip