search

xcube-dev / xcube

xcube is a Python package for generating and exploiting data cubes powered by xarray, dask, and zarr.
https://xcube.readthedocs.io/
MIT License
198 stars 18 forks source link

`data_store.write_data()` with `replace=True` to s3 bucket corrupts cube #761

Open AliceBalfanz opened 1 year ago

AliceBalfanz commented 1 year ago

Describe the bug When writing a cube with data_store.write_data(ds, output_id, replace=True) by using the s3 data store, the final cube is corrupt.

To Reproduce Steps to reproduce the behavior:

  1. use xcube datastore for s3 to write a cube into s3 bucket
  2. modify cube, and write it again to the same location by using parameter "replace=True"
  3. view cube, it is corrupt - contains no metadata files

Expected behavior I would expect, that xcube first removes all files, and then writes the new cube into the same location as the previous one. Making sure no files from the old one corrupt the new cube as well as making sure the metadata exists.

forman commented 1 year ago

Please make sure it is not a permission issue.

The functionality is well-tested, both by unit tests and in several projects.