search

computational-cell-analytics / micro-sam

Segment Anything for Microscopy
https://computational-cell-analytics.github.io/micro-sam/
MIT License
311 stars 35 forks source link

Multiscale image layer support #590

Open constantinpape opened 2 months ago

constantinpape commented 2 months ago

We currently don't support multiscale image layers and the goal of this PR is to address this.

If a user selects a multiscale image layer for embedding computation they will now be prompted to select the level to be used:

mip-embeddings

It already works for level zero, but does not work correctly for other levels yet.

What's left to do:

  1. Scale the prompts if the embeddings are computed on a level > 0. Prompts are given in the coordinate system w.r.t. scale 0, which does not match the embeddings. This is relatively simple: compute the scaling factor when computing the embeddings, add them to the state, scale prompt coordinates with it in the segmentation functions.
  2. Decide what to do for the label layer for level > 0. Currently a label layer would be create with the corresponding shape, so the segmented objects would not match the image data (which is displayed in the coordinate system of level 0). There are two options for solving this:
    • Scale the label layer with the same scale factor as in 1. (Can directly use scale functionality of the napari layer)
    • Create a multiscale layer for the labels as well (I think this is overkill and would complicate things, I would rather go with the prior option, just mentioning this here for completeness)
constantinpape commented 1 month ago

This is working now for the 2d annotation tool. May still need some changes to make it work for 3d data.

constantinpape commented 1 month ago

3D is a bit more complicated. Will revisit this next week.