Reduce code duplication with `funlib.persistence`

funkelab / gunpowder

A library to facilitate machine learning on multi-dimensional images.

https://funkelab.github.io/gunpowder/

MIT License

78 stars 56 forks source link

Reduce code duplication with `funlib.persistence` #206

Open adjavon opened 7 months ago

adjavon commented 7 months ago

Many of the specifics of reading Zarr files are already handled in fun lib.persistence. It also already generalizes to other sources of array data. Goal is to generalize the ZarrSource by making it use an Array object.

[ ] Change ZarrSource into an ArraySource that uses persistence in the background
[ ] Have the ArraySource read N5, numpy arrays, etc.
[ ] Deprecate H5LikeSource
[ ] Deprecate KLBSource

pattonw commented 7 months ago

There is a little ArraySource in the tests/helpers.py. Fairly simple to make with any gunpowder Array class that can be built from numpy arrays and I think most things that resemble numpy arrays? I haven't tested that. It might be worth while to depricate the gunpowder Array class and just use the funlib.persistence Array class, but they are a bit different with the whole ArraySpec for gunpowder.

adjavon commented 7 months ago

What we were thinking about was more specifically related to the out-of-memory arrays like zarr datasets (or numpy saved files, rather than arrays). So for things like ROI-based slicing etc.

This is more of a self-reminder for something we're thinking of working on next week which you should probably be a part of @pattonw! I'll reach out

pattonw commented 2 months ago

This is now available via the ArraySource. It takes a funlib.persistence.Array and an ArrayKey. old HDF5LikeSource not yet depricated. I'll depricate in the next patch release and remove in the next minor release. old HDF5LikeWrite nodes should still be depricated and replaced with an ArrayWrite node. TODO for next patch release