search

xtensor-stack / xtensor-zarr

Implementation of the Zarr core protocol (version 2 and 3) based on xtensor.
http://xtensor-zarr.readthedocs.io
BSD 3-Clause "New" or "Revised" License
39 stars 10 forks source link

generating .zr3 files for zarr_implementations #38

Open grlee77 opened 3 years ago

grlee77 commented 3 years ago

I saw that a recent PR by @davidbrochart (https://github.com/zarr-developers/zarr_implementations/pull/23) added xtensor support to zarr_implementations. I was interested in extending the data generation and testing there to include the zarr v3 format as well.

I tried this via adding the following lines to the main.cpp file in that PR:

    // zarr v3 raw
    const std::string hier_path_zr3 = "../../../data/xtensor_zarr.zr3";
    auto h_zr3 = create_zarr_hierarchy(hier_path_zr3, "3");
    xzarr_create_array_options<xio_binary_config> raw_options_zr3;
    raw_options_zr3.fill_value = 0;
    zarray zr3_raw = h_zr3.create_array("/raw", shape, chunk_shape, "u1", raw_options_zr3);
    noalias(zr3_raw) = img;

    // zarr v3 gzip
    zarray zr3_gzip = h_zr3.create_array("/gzip", shape, chunk_shape, "u1", gzip_options);
    noalias(zr3_gzip) = img;

    // zarr v3 zlib
    zarray zr3_zlib = h_zr3.create_array("/gzip", shape, chunk_shape, "u1", zlib_options);
    noalias(zr3_zlib) = img;

    // zarr v3 blosc
    auto g_blosc_zr3 = h_zr3.create_group("/blosc/");
    zarray zr3_blosc_lz4 = h_zr3.create_array("/gzip", shape, chunk_shape, "u1", blosc_options);
    noalias(zr3_blosc_lz4) = img;

This compiles fine, but I get a segfault at runtime on the first attempt to assign img to one of the zr3 arrays (i.e. no crash occurs if I comment out all of the assignment lines like noalias(zr3_raw) = img;).

The error I see is as follows:

make[4]: Entering directory '/media/lee8rx/data/Dropbox/Dropbox/Quansight/public/zarr/zarr_implementations/generate_data/xtensor_zarr/build'
[ 50%] Building CXX object CMakeFiles/run_xtensor_zarr.dir/src/main.cpp.o
[100%] Linking CXX executable run_xtensor_zarr
make[4]: Leaving directory '/media/lee8rx/data/Dropbox/Dropbox/Quansight/public/zarr/zarr_implementations/generate_data/xtensor_zarr/build'
[100%] Built target run_xtensor_zarr
make[4]: Entering directory '/media/lee8rx/data/Dropbox/Dropbox/Quansight/public/zarr/zarr_implementations/generate_data/xtensor_zarr/build'
Scanning dependencies of target run
make[4]: Leaving directory '/media/lee8rx/data/Dropbox/Dropbox/Quansight/public/zarr/zarr_implementations/generate_data/xtensor_zarr/build'
make[4]: Entering directory '/media/lee8rx/data/Dropbox/Dropbox/Quansight/public/zarr/zarr_implementations/generate_data/xtensor_zarr/build'
terminate called after throwing an instance of 'std::runtime_error'
  what():  write: failed to open file ../../../data/xtensor_zarr.zr3/data/root/raw/c0/0/0

If I look in the generated ../../../data/xtensor_zarr.zr3 folder, the expected metadata has been generated, but there is no data directory. Is this expected to work at this point and/or am I doing something incorrectly? Thanks for any help you can provide.

davidbrochart commented 3 years ago

Thanks for reporting @grlee77, I'm looking into it.