Nicholas-Schaub
commented
2 years ago Hey @ctrueden
This is going to be a wall of text because there is a lot of history here. I'm happy to set up a quick meeting to discuss if you're available/willing. In summary, bfio predates the existing functionality in AICSImageIO, predates PyImageJ, and was originally created as a wrapper around python-bioformats to circumvent/modify a number of the issues present in that package. We have discussed merging with AICSImageIO, and what is likely going to happen is that we are going to merge some of our code into their codebase, but we are going to maintain bfio separately for a variety of reasons I'll discuss below.
bfio was originally created as a wrapper around python-bioformats in early 2019 (existing as a folder in polus-plugins) to support our data science teams create interoperable plugins for an NIH funded cloud computing project called Polus, which includes the NIST open source project WIPP. WIPP uses .ome.tif as it's standard file format, specifically using tiled chunks to improve performance. At the time, python-bioformats did not handle chunked reading/writing of data well, and ome model metadata handling was not sufficient for what we needed. We attempted to use tifffile, which was excellent for reading and more performant, but we had similar issues saving data at that time. AICSImageIO did not have bioformats as a part of the source at that time, and had limited functionality relative to where it's at now.
Over time, the biggest bottleneck for bfio was slow read/write times to/from .ome.tif, and we were considering creating a new zarr based file format. What we ended up doing is rewriting the codebase for bfio, which is when the code was migrated from polus-plugins to here. Part of that was creation of an optimized ome.tif class that modified code from tifffile to improve read/write times and also allow us to maintain metadata properly. When we did that, we also dissociated from python-bioformats and created our own bioformats bindings through jpype, since we found a large number of issues working with various file formats through python-bioformats. Essentially, python-bioformats had custom code for class selection that caused issues for a number of supported formats. What bfio does now is use bioformats as a catch-all, and we have custom ome tiff and ome zarr classes for reading/writing of data. If we do not have a custom implementation for a specific format, we try to load a file with bioformats. We explicitly try to avoid using Bioformats as a general rule, because we have found the overhead to be pretty high relative to our dedicated readers and writers.
We have been in discussions with the AICSImageIO team, and even considered merging this work with their repo. We are still going to create bindings for our OME Tiff reader/writer classes because they seem to be more performant than what they currently have. However, we have decided to continue maintaining bfio for a variety of reasons. One of which is that bfio 2.X is relatively polymorphic with numpy, and treats images as memory mapped arrays (something that AICSImageIO also does). This makes adaptation of an existing image processing algorithm easier because we can just replace a numpy array with a bfio object. Also, there is relatively little boilerplate code required to use bfio, lowering the burden on our developers to implement new algorithms.
So, in summary, we found python-bioformats to be a problematic implementation of bioformats, AICSImageIO was not where it is now when we first started (or we may have used that), and PyImageJ did not exist. If you take a look at our polus-plugins repo, you will actually see we have a large set of projects involved in scraping ImageJ Ops using PyImageJ, so we have pretty good knowledge of what it is capable of doing and are closely watching that (particularly the macro functionality that has an issue with the legacy window manager). We have kept a close eye on AICSImageIO and have been in contact with that team, and plan on contributing. However, at the end of the day, the developer usability of our package and the optimized tools specifically for formats in WIPP is what is holding us back from jumping completely on board.
My 2 cents is that python-bioformats should go in the wastebasket if they are considering what to focus on. Not because I am lacking in appreciation of what they accomplished at all, because at the time we first started working on bfio that looked like the best approach from our end. I just find jpype to be a superior and more natural interface to Java, and I think any Python implementation using python-javabridge is going to carry a lot of technical debt. We have also been following scyjava, but I think any tool that directly interacts with bioformats is going to have subpar performance. The data we receive from NIH is generally large, and single images frequently are too large to fit into memory.
We have discussed bfio with the Bioformats team at some ngff meetings. I know they are aware of our work, and I think you will find that some of the issues here are from their team. I think it's a good suggestion to submit a PR to their repo. We have done relatively little advertising about bfio, but we will be changing that soon.
ctrueden
The Bio-Formats documentation lists three ways to interface Bio-Formats from Python. See here:
https://docs.openmicroscopy.org/bio-formats/6.8.0/developers/python-dev.html
Those three ways are:
The AICSImageIO and scyjava approaches use JPype. The python-bioformats approach uses python-javabridge, but the COBA team (Broad Imaging Platform & LOCI/Eliceiri lab) is currently investigating either retiring python-bioformats or migrating it to be built on scyjava also.
My question is: what were the motivations behind the development of bfio versus adopting one of these other existing projects?
Totally understandable if it was done naively, or before these other projects were mature enough. That happens all the time in tech of course. But maybe now could be a good time to step back and double check project requirements, to evaluate whether it might be possible to reconcile these overlapping libraries?