bioio-devs / bioio-ome-zarr

A BioIO reader plugin for reading Zarr files in the OME format.
BSD 3-Clause "New" or "Revised" License
0 stars 2 forks source link

(Not a fork) Bugfix: read zarr from s3 #17

Closed pgarrison closed 4 months ago

pgarrison commented 5 months ago

Same as #15 but not from a fork

This pull request resolves #13

Description of Changes

Additional changes

Testing

  1. Added a test against a public S3 bucket to this repo (it passes on my machine).
  2. Created a new project with the following dependencies and ran the following integration test (this used to error). The integration test requires AWS credentials to be configured (e.g., in the ~/.aws directory).
    from bioio import BioImage
    path = "s3://allencell/aics/nuc_morph_data/data_for_analysis/baseline_colonies/20200323_09_small/raw.ome.zarr"
    image = BioImage(path)
    print(image.get_image_dask_data())
    # Dependencies
    "bioio-ome-zarr @ git+https://github.com/bioio-devs/bioio-ome-zarr.git@bugfix/read-zarr-from-s3",
    "bioio @ git+https://github.com/bioio-devs/bioio.git@main",
    "bioio-base @ git+https://github.com/bioio-devs/bioio-base.git@main"
pgarrison commented 5 months ago

The new test is failing but I have not figured out why. I can't reproduce the issue locally

toloudis commented 5 months ago

The new test is failing but I have not figured out why. I can't reproduce the issue locally

Did you see this warning in the build logs?

UserWarning: Your installed version of s3fs is very old and known to cause
  severe performance issues, see also https://github.com/dask/dask/issues/10276

Maybe compare your local version of s3fs with the one used in the automated build?

pgarrison commented 5 months ago

Figured it out -- the underlying ome-zarr packages doesn't support passing { anon: True } to fsspec. Filed an issue https://github.com/ome/ome-zarr-py/issues/369

pgarrison commented 5 months ago

https:// urls work with this plugin, but do not integrate correctly with bioio-base