List evaluation metrics

[ebrahimebrahim]

list evaluation metrics that we will use to evaluate pairwise registration methods

we currently have

• fiber distance hausdorff like thing

other things:

• number of folds (jacobian det)
• runtime
• tract dice

[jamesfishbaugh]

Here are the most commonly used metrics.

1) Tract distance (mean closest fiber distance) [1]

2) Tract dice (overlap of binary tract segmentation masks) [1]

3) Tract volumetric similarity (for a non-overlap measurement of binary tract segmentation masks) [2]

4) Jacobian of deformation field (percentage of voxels with non-positive determinant) [1]

5) Runtime

Other possibilities which are commonly used in template based evaluations (which we will need eventually as well)

6) FA profiles along fiber bundles (compare average FA profiles along fiber tracts before and after registration) [3]

7) Cosine similarity (better registration indicative of smaller intersection angle between tracts) [4]

Refs:

[1] Zhang, Fan, William M. Wells, and Lauren J. O’Donnell. "Deep diffusion MRI registration (DDMReg): a deep learning method for diffusion MRI registration." IEEE transactions on medical imaging 41.6 (2021): 1454-1467.

[2] Taha, Abdel Aziz, and Allan Hanbury. "Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool." BMC medical imaging 15 (2015): 1-28.

[3] Wang, Yi, et al. "DTI registration in atlas based fiber analysis of infantile Krabbe disease." Neuroimage 55.4 (2011): 1577-1586.

[4] Wang, Yi, et al. "Evaluations of diffusion tensor image registration based on fiber tractography." Biomedical engineering online 16 (2017): 1-20.

[ebrahimebrahim]

Do we want to do all of these? Is this list exhaustive?

6. Cosine similarity (better registration indicative of smaller intersection angle between tracts) [4]

They say

\cos\alpha = \frac{1}{|F||G|} \sum_i \sum_j F_i\cdot G_j

but I am confused what is meant by $F_i\cdot G_j$. They say $F$ and $G$ are bundles, so I guess $F_i$ and $G_j$ are individual fibers? What is the dot product between two fibers? Maybe they explain it and I am being lazy :smile:

---

Other than that the list looks good. I would just add runtime

[jamesfishbaugh]

Do we want to do all of these? Is this list exhaustive?

I would say the first 5 are the priority for pairwise registration. We should ask Martin about how he feels about 6 and 7 in the context of pairwise evaluation. The list isn't exhaustive but these are the main measurements I saw repeated across multiple studies.

What is the dot product between two fibers?

It was also not immediately clear to me what they mean by the dot product between two streamlines. This is another question we can ask Martin as the concept also shows up in his work.

The motivation to include tract volumetric similarity is based on a study I did years ago about rankings for grand challenges. The idea is that a variety of metrics based on different concepts give a more complete description of accuracy. Also, dice and volumetric similarity are not highly correlated as many other metrics are so they are reasonable and common choices.

[ebrahimebrahim]

Can you explain your rationale for including tract volumetric similarity in the list? A few excerpts from [2]:

In the following are some of the requirements that could be put on the segmentation algorithms.

• Alignment is important: When the requirement is the location (general alignment) of the segment rather than the boundary delimitation. In this case, the volume based metrics are not a good choice.
• ...
• Volume is important: Sometimes the magnitude of the segmented region is of more importance than the boundary and the alignment. Here, algorithms should segment region to have a volume as near to that of the true segment as possible. The volumetric similarity VS is recommended.

When the objective is to evaluate the general alignment of the segments, especially when the segments are small (the overlap is likely small or zero), it is recommended to use distance based metrics rather than overlap based metrics. The volumetric similarity (VS) is not suitable in this case.

When the segmented volume is of importance, volumetric similarity and overlap based metrics are recommended rather than distance based-metrics.

[ebrahimebrahim]

1. Tract distance (mean closest fiber distance) [1]
2. Tract dice (overlap of binary tract segmentation masks) [1]
3. Tract volumetric similarity (for a non-overlap measurement of binary tract segmentation masks) [2]
4. Jacobian of deformation field (percentage of voxels with non-positive determinant) [1]
5. Runtime

Other possibilities which are commonly used in template based evaluations (which we will need eventually as well)

6. FA profiles along fiber bundles (compare average FA profiles along fiber tracts before and after registration) [3]
7. Cosine similarity (better registration indicative of smaller intersection angle between tracts) [4]

For now we will focus on (1) through (5) as metrics. I didn't realize (6) and (7) were suggested as suitable for template based evaluation.