SAM++: Enhancing Anatomic Matching using Semantic Information and Structural Inference

[Wangrui-berry]

In SAM++, for the semantic branch, there should be a label (e.g., lesion, kidney). As a result, is it supervised or not self-supervised? What's more, query inference using fixed-points based structural inference seems also an average of the mapping of 3 fixed points, isn' it? (Am I understand right?)

[viggin]

The original SAM is purely self-supervised, but we found it may be confused when two organs are adjacent and without clear borders, so we added semantic organ masks as supervision to enhance the embedding. Now it has supervision, but I think it is not "supervised" in the traditional way, because our goal is universal landmark detection, but we did not use landmark supervison. Maybe kind of weakly supervised.

[viggin]

About the fixed-points based structural inference, please refer to our paper: https://arxiv.org/pdf/2306.13988.pdf and our code: https://github.com/alibaba-damo-academy/self-supervised-anatomical-embedding-v2/blob/main/tools/demo_semantic_stable_points.py We first use the mapping of fixed points to estimate a transform matrix, then use it to transfer the point.

[aa1234241]

In SAM++, for the semantic branch, there should be a label (e.g., lesion, kidney). As a result, is it supervised or not self-supervised? What's more, query inference using fixed-points based structural inference seems also an average of the mapping of 3 fixed points, isn' it? (Am I understand right?)

The three fixed points displayed in the figure are used for demonstration purposes only. If we select a 5x5x5 cubic region around the template point, the total number of points is 125, and we can expect to obtain dozens of fixed points. These fixed points are employed to estimate the transform matrix $\mathbf{T}$. Finally, the query point location is computed from the transform matrix $q=Tt$.