Python bindings

[aschampion]

To provide a lower installation barrier, more N5-specific python access to N5 than the excellent z5py, it would be nice to have python bindings via rust-n5 than supports:

• [ ] dataset creation
• [ ] attribute access
• [ ] numpy array read/write

This would be implemented in a separate, dependent python binding crate (possibly in this repo with a cargo workspace), but is tracked here.

Two main python binding libraries exist for rust (comparison):

• PyO3
• rust-cypython

On first examination I tend to prefer PyO3 (see this rather nice example). The rust numpy binding library currently uses rust-cpython, but plans to migrate (and has an in-progress PR) to PyO3 once it supports stable rust. Will revisit this issue once that has happened.

cc @clbarnes

[clbarnes]

Do you envision a numpy-like interface to datasets or would you be more inclined towards going a separate way, like the reference implmentation of zarr? Catching enough slicing and broadcasting edge cases to be usefully numpy-compatible is awkward (although in some cases, already implemented in z5).

[aschampion]

@clbarnes and I discussed this offline, but the reasonable minimum here is read/write access to dimension-complete, contiguous, non-strided slices for now, and only supporting broadcasting for scalar values. The numpy access will work via the existing ndarray access, so will not be zero-copy but should be reasonably fast.

[aschampion]

rust-numpy has migrated to PyO3; their release is blocked on the next release (0.3) of PyO3, but that's nearing. So that blocker has been removed.

[clbarnes]

@pattonw expressed some interest in this.

For the use of anyone who might attempt it, some of the sticking points I came across working on z5py:

• Implementing __getitem__ in such a way as to approximate numpy's API is tricky. My attempts are here: https://github.com/constantinpape/z5/blob/master/src/python/module/z5py/shape_utils.py , don't implement striding, and have a couple of other features numpy doesn't.
• Attribute read/writes should be atomic as possible, and do their best to lock the file for the duration. The attributes object should subclass MutableMapping. My attempts are here https://github.com/constantinpape/z5/blob/master/src/python/module/z5py/attribute_manager.py but are not ideal - it would be better to have rust do the serialisation/deserialisation and pass PyObjects back up, and for attribute writes from python to not have to read the entire dict first. I think it's fine for reads to pass the entire dict from rust up to python, though, because the alternative is implementing the entire Mapping API in rust, and because it's deserialising the whole thing anyway (not necessarily strictly true but close enough)

In the unlikely event that we wanted anything more than an MVP, https://github.com/Quansight-Labs/uarray seems to be an attempt to formalise the numpy ndarray interface for non-numpy projects, including explicit handling of ufuncs and the like.

[aschampion]

This is fulfilled by https://github.com/pattonw/rust-pyn5