search

mapmanager / MapManagerCore

A Python library with the core functionality of MapManager
0 stars 0 forks source link

Development maps/timeseries will have equal image shape/dtype per timepoint #17

Closed cudmore closed 1 month ago

cudmore commented 2 months ago

Start using an actual multi-timepoint map for development

For now, during development, we will assume a map/timeseries will have equal shape/dtype image(s).

To build a map from an Igor-Map we just need to add empty (0) slices to make all images in the timeseries have the same shape.

See /sandbox/import_mm_igor.py for an import function.

To run this locally and generate a map, you need to make a local copy of the MapManager-Data folder.

Make sure MapManager-Data folder is at the same level (next to) the MapManagerCore folder.

Then run from the MapManagerCore folder

python sandbox/import_mm_igor.py

This will generate a mmap file in the sandbox folder (make sure you do not push this!)

In the sandbox/import_mm_igor.py file, you can either generate a single timepoint mmap or a timeseries (8 session) mmap by tweaking the oneTimepoint variable. In this example it will generate a single timepoint mmap from Igor timepoint 0. Use oneTimepoint = 5 to generate a single timepoint mmap from Igor timepoint 5 (for example).

oneTimepoint = 0  # set to None to make a map of numSessions

An actual map/timeseries mmap zarr file opens a can of worms!

Once I started using a multi timepoint map (from zarr), lots of things got super slow. Will post more issues with details.

In particular, fetching spine annotation info using map[:] got unusably slow. Also, fetching left/right radiu with map.segments["segmentLeft"] got slow.

Hint hint hint. I would like to see all our computed values of both spines and lines get saved to the zarr file. On next load, in a GUI viewer, there is no compute needed. AFAIK this is not the way it is implemented, our computed values (spine intensity, line left/right) are not saved in the file but computed during runtime.

Production Version

In the production version we can not assume each image stack in each timepoint to have the same shape. They should be able to have different shapes (will always have different z-slices), different dtypes(s), etc.

We can hold off implementing this for now.

Suhayb-A commented 2 months ago

The current system saves all lazily computed columns so future loads will not require a wholesale recompute. I am working on a patch that could speed some things up. There are a lot of low-hanging performance improvements. Note that we have yet to make anything parallel, this can be done by injecting code into the query library. It will also be rare for the entire dataset to require computation since the values will be incrementally computed as needed and saved to a file.

You narrowed down the issue to the calculation of the left and right segments, which makes sense. The loaded lines are probably dense and need to be optimized. Furthermore, I am considering deprecating the left and right calculations that include the z index, instead, I am considering providing an optimized way to get pre-clipped/filtered paths similar to how the layers system retrieved the left and right segments.

As for having variations in the shape of the image, the latest version which I am still fixing/optimizing supports timepoints with different shapes. Awesome to see the progress on igor pro loader!