Reserved values for uint64

[pgunn]

Right now it looks like uint64 support for n5 files has a value limitation compared to other languages (Java, for examples) - it is not possible to store values that get closer than 1024 to UINT64_MAX

>>> fh['seg'][2,2,2] = np.iinfo(np.uint64).max - 1024
>>> fh['seg'][2,2,2]
18446744073709549568
>>> fh['seg'][2,2,2] = np.iinfo(np.uint64).max - 1023
>>> fh['seg'][2,2,2]
0

(this creates problems when some meaningful magic values in imaging formats are up there and we can't assign them)

[constantinpape]

I can reproduce it (for both n5 and zarr file formats) and that's a weird one! Unfortunately I don't have any time looking into this for the foreseeable future. A contribution that solves this would be highly appreciated.

As a first pointer for how to solve it:

• one should test if this issue also occurs when using only c++ (to see if it an issue in the c++ backend or if it happens in the python/c++ glue code)
• if it's in the glue code: this is most likely in upstream issue in xtensor-python or pybind11
• if it happens in c++: one would need to check where exactly it happens when handling the data buffer (e.g. this is a good starting point for checking this: https://github.com/constantinpape/z5/blob/master/include/z5/filesystem/dataset.hxx#L36)

[pgunn]

I meant to spend a bit of time trying to figure this out this weekend, but so far I'm having trouble getting the software to build locally (followed the instructions, it seems to have tripped over some simd related stuff in xtensor headers).

/home/pgunn/miniconda3/envs/z5/include/xtensor/xtensor_simd.hpp:65:18: error: ‘set_simd’ has not been declared in ‘xsimd’
   65 |     using xsimd::set_simd;

Knowing C++, this could as easily be a compiler version problem or any number of other things. I'll see if I can find another one of my systems where the build has more luck.

[constantinpape]

(followed the instructions, it seems to have tripped over some simd related stuff in xtensor headers).

That might be some version conflict in xtensor and xsimd. In case you have used one of the environments from https://github.com/constantinpape/z5/tree/master/environments/unix : It might help to also pin the xsimd version in the env. I have xsimd 8.0.5 in my local dev environment, so pinning xsimd>=8,<9 might fix the issue.

[pgunn]

Sadly, same thing. Debugging C++ build errors is not my forte.

[ 14%] Building CXX object src/python/lib/CMakeFiles/_z5py.dir/z5py.cxx.o
In file included from /home/pgunn/miniconda3/envs/z5-py36/include/xtensor/xstorage.hpp:22,
                 from /home/pgunn/miniconda3/envs/z5-py36/include/xtensor/xbuffer_adaptor.hpp:21,
                 from /home/pgunn/miniconda3/envs/z5-py36/include/xtensor-python/pyarray.hpp:17,
                 from /home/pgunn/src/z5/src/python/lib/z5py.cxx:8:
/home/pgunn/miniconda3/envs/z5-py36/include/xtensor/xtensor_simd.hpp:65:18: error: 'set_simd' has not been declared in 'xsimd'
   65 |     using xsimd::set_simd;
      |                  ^~~~~~~~
/home/pgunn/miniconda3/envs/z5-py36/include/xtensor/xtensor_simd.hpp:66:18: error: 'load_simd' has not been declared in 'xsimd'
   66 |     using xsimd::load_simd;
      |                  ^~~~~~~~~
/home/pgunn/miniconda3/envs/z5-py36/include/xtensor/xtensor_simd.hpp:67:18: error: 'store_simd' has not been declared in 'xsimd'
   67 |     using xsimd::store_simd;

[constantinpape]

Can you share the version of xsimd, xtensor and xtensor-python in the env where this is happening?

[pgunn]

https://pastebin.com/32AUw7Ra

[constantinpape]

I will try setting up an env later and see if I can reproduce it. Which OS are you using?

[pgunn]

I'm on Debian 12 on x86_64.

[constantinpape]

The dev environment was very outdated. I have updated it and also simplified it because it's not necessary to have the fixed envs for python versions any more. It works on my laptop with these changes (Ubuntu 22.04). See #219.

[pgunn]

It builds now! I'll hopefully be able to spend some time digging on this issue now. Thanks!