search

brisvag / blik

Python tool for visualising and interacting with cryo-ET and subtomogram averaging data.
https://brisvag.github.io/blik/
GNU General Public License v3.0
23 stars 8 forks source link

Use dask #120

Closed brisvag closed 1 year ago

brisvag commented 2 years ago

The first time we introduced lazy loading we tried with dask.delayed, but ultimately were trying to manually load the images when needed. Napari has pretty good handling of dask delayed now, and we should be able to simpoly load all the images as delayed and let napari handle everything!

Even better, this lets us use do cool thing like load trhough ssh :D

alisterburt commented 2 years ago

yes! Will this replace your lazy loading/affect your metadata capture stuff?

alisterburt commented 2 years ago

extra thoughts - are you suggesting we use something other than delayed? What juan showed was da.from_zarr, we can do da.from_delayed and pass that to napari, is that what you're suggesting?

brisvag commented 2 years ago

yes! Will this replace your lazy loading/affect your metadata capture stuff?

Yes, we still need to read headers! We could actually write an mrc reader package (or contribute to the original) so that this is handled on that side.

are you suggesting we use something other than delayed?

Not sure what you mean; yeah, I was thinking da.from_delayed, and we just have to construct the delayed first. We can even use the memory map version for mrcs, so 2D slices are even more memory efficient :)

alisterburt commented 2 years ago

mrc reader package for napari which returns a dask array and header info you mean? I've been meaning to add one to AICSImageIO based on Talley's recommendation of moving away from millions of isolated reader plugins

all sounds good on the delayed front!

brisvag commented 2 years ago

exactly! Sure, that sounds great :)