Atlas based segmentation with AMU7T template

[Nilser3]

Description

AMU7T is a 3D template created in 2 contrasts acquired at 7T:

• qT1-> from 26 subjects
• t2s -> from 72 subjects an atlas of 50 labels is associated (WM parcels from Levi et al., NeuroImage (2015)., and GM parcels derivated from _Hausman, _L. Atlases of the Spinal Cord and Brainstem and the Forebrain. (Thomas, 1962).__ AMU7T was created in the space of PAM50 (related in spinalcordtoolbox/PAM50#14 ), and in a resolution of 0.17x0.17x0.5mm.

@jcohenadad this issue is to discuss how to automatically segment based on this new high resolution template/atlas through SCT:

here is a t1q 7T image that was automatically segmented using Sct registration (a) and a method with Optimized parameters (b).

From ISMRM2023

Sct registration (a)

# Register 7T MP2RAGE image  (t1q) to PAM50
sct_register_to_template -i t1q.nii.gz -s t1q_sc.nii.gz -l landmarks_t1q.nii.gz -c t2s -v 1 
# Register of  AMU7T_50_labels.nii.gz to to subject space
sct_apply_transfo  -i PAM50/AMU7T/atlas/AMU7T_50_labels.nii.gz -d t1q.nii.gz -w warp_template2anat.nii.gz -o tract_from_t1q_wm.nii.gz -x nn

Optimized parameters (b)

# Register 7T MP2RAGE image  (t1q) to PAM50
sct_register_to_template -i t1q.nii.gz -s t1q_sc.nii.gz -l landmarks_t1q.nii.gz -c t2s -v 1 

# Register of AMU26_qt1.nii.gz and AMU7T_wm.nii.gz to subject space using warps 
antsApplyTransforms -d 3 -i PAM50/AMU7T/template/AMU26_qt1.nii.gz -o qT1_AMU26_ants.nii.gz -t warp_template2anat.nii.gz  -r t1q.nii.gz
antsApplyTransforms -d 3 -i PAM50/AMU7T/template/AMU7T_wm.nii.gz -o wm_AMU26_ants.nii.gz -t warp_template2anat.nii.gz  -r t1q.nii.gz

# Adjustment of the registration in the subject space with a SyN step
antsRegistration --verbose 1 --dimensionality 3 --float 0 --collapse-output-transforms 1 --output [wm_t1q_s,wm_t1q_sWarped.nii.gz] --interpolation Linear --transform SyN[0.2,3,0] --metric CC[t1q.nii.gz,qT1_AMU26_ants.nii.gz,0.2,2] --metric CC[t1q_wm.nii.gz, wm_AMU26_ants.nii.gz,0.8,2] --convergence [100x75x50,1e-6,50] --shrink-factors 4x2x1 --smoothing-sigmas 2x1x0vox -g 1x1x0 -x [t1q_sc.nii.gz, t1q_sc.nii.gz]

# Register of  AMU7T_50_labels.nii.gz to to subject space using 2 warps 
antsApplyTransforms -d 3 -i PAM50/AMU7T/atlas/AMU7T_50_labels.nii.gz  -o tract_from_t1q_wm_ants.nii.gz   -t wm_t1q_s0Warp.nii.gz -t warp_template2anat.nii.gz -r t1q.nii.gz -n MultiLabel

[joshuacwnewton]

this issue is to discuss how to automatically segment based on this new high resolution template/atlas through SCT:

Given that the goal of this issue is to create a "pure SCT" way to do step (b), and that this will involve updating sct_warp_template, I think it might be best to transfer this issue to the SCT repo for organizational purposes.

[joshuacwnewton]

In our July 6th meeting, we asked "Should AMU7T be integrated into the PAM50 repo, or should it be kept separate?"

I don't think we quite came to a definitive answer during the meeting, so here are my thoughts:

---

Looking at the associated AMU7T PR (https://github.com/spinalcordtoolbox/PAM50/pull/14), I notice that the AMU7T template has a similar structure to the PAM50 template as a whole:

• AMU7t:
• PAM50:

In other words, AMU7t is more like an alternate version of PAM50 (with alternate atlas/ and template/ folders), rather than a distinct subfolder within the PAM50 template (like histology/ was).

---

Given this, I am kind of leaning towards keeping AMU7t separate from PAM50:

• Users would "opt-in" by selecting the template using the -t (template) options of sct_register_to_template and sct_warp_template, which would allow us to re-use the existing template/atlas warping code.
• Pros: Much less complicated implementation (no need for new separate options, no need for multiple nested atlas or template folders, can reuse existing functionality [which is kind of the whole purpose of having a -t option!]).
• Cons: Users would have to run sct_warp_template twice: Once to warp the PAM50 template/atlas and once to warp the AMU7t template/atlas. Also, users have to "opt-in", rather than immediately getting the warped AMU7t via the default options, which might make the AMU7T template/atlas harder for users to discover.

[jcohenadad]

Excellent arguments @joshuacwnewton. Thank you for bringing that up. In light of these arguments, I agree that it makes more sense to have AMU7t as a separate template (despite it being within the same space as the PAM50 template).

[joshuacwnewton]

I agree that it makes more sense to have AMU7t as a separate template

Great! In that case, I would say the next steps are:

• [x] Close the current PR on the PAM50 repo
• [x] Create a new repo containing the current files for the AMU7T template
• [x] Try out the sct_register_to_template and sct_warp_template commands with the -t option pointing at the AMU7T folder, to see if there are any issues.
• [ ] After we make sure that the -t option works well, then opening a PR in the SCT repo to add the AMU7T template to sct_download_data

As well, there is also the separate observation you noticed during our last SCT meeting:

• [ ] Improving the alignment for the "AMU26_qt1.nii.gz" version of the AMU7T template (the blue .GIF in the PAM50 PR)

cc: @Nilser3

[Nilser3]

Thanks for your comments @joshuacwnewton @jcohenadad

I closed the PR, and created a repo for template_AMU7T

I should point out that AMU7T is a template cropped in XY plane of 19.3 x 15.8 mm and along the Z axis from C1 to C7 approximately. Here a visualization with the levels of PAM50

Some Experiences of registration:

Here 7T T1q image + SC mask with which we are going to experiment ( r = 0.3x0.3x5.0):

Exp_1. Registration using PAM50:

sct_register_to_template -i image.nii.gz -l landmarks_pal1.nii.gz -c t1

Time: 98 sec

Exp_2. Registration using original AMU7T:

sct_register_to_template -i image.nii.gz -l landmarks_pal1.nii.gz -c t1 -t ../../../template_AMU7T/

Comment: Edge problems and loss of information at different levels.

Time: 37 sec

Exp_3. Registration using AMU7T with a zero padding on the XY axis (to get the same size as PAM50 from C1 to C7 )

sct_register_to_template -i image.nii.gz -l landmarks_pal1.nii.gz -c t1 -t ../../../template_AMU7T/

Comment: a partial loss of information in C8 (comparing with Exp_1)

Time: 306 sec

Exp_4. Registration using AMU7T with a zero padding in the XY and Z axis (C1 to T1):

sct_register_to_template -i image.nii.gz -l landmarks_pal1.nii.gz -c t1 -t ../../../template_AMU7T/

Comment: Results similar to exp_3

Time: 357 sec

Exp_5. Registration using AMU7T with a zero padding on the XYZ axis (C1 to T1) + using SC mask of PAM50 (at the same resolution and dimension):

sct_register_to_template -i image.nii.gz -l landmarks_pal1.nii.gz -c t1 -t ../../../template_AMU7T/

Comment: less deformation is observed in C1 and C8, with more resemblance to Exp_1

Time: 356 sec

Use of warps

a) sct_warp_template works for template_AMU7T/atlas/... labels

sct_warp_template -d image.nii.gz -t ../../../template_AMU7T/

Comment: approximate segmentation of the CSF contour.

b) sct_apply_transfo foratlas/AMU7T_50_labels.nii.gz

sct_apply_transfo -i ../../../template_AMU7T/atlas/AMU7T_50_labels.nii.gz -d image.nii.gz -w warp_template2anat.nii.gz -x nn Comment: atlas-based segmentation with errors

The option -t is still not working well, I am trying to use a sct_register_multimodal between anat2template from Exp_5 and /template_AMU7T/template/AMU26_qt1.nii.gz (using WM masks) to adjust the registration,

I have also seen that in the sct_register_to_template they do a resampling to 1mm isotropic, so I will also try to improve the registration in this way.

[jcohenadad]

We should redirect as many information/context regarding the AMU7T template (eg this very informative comment https://github.com/spinalcordtoolbox/template_AMU7T/issues/4#issuecomment-1634658886 which will be forgotten if not placed in the appropriate repos) as possible in the dedicated repos (https://github.com/spinalcordtoolbox/template_AMU7T) instead of here. @joshuacwnewton can you please help @Nilser3 do some cleaning / transfer of issues, etc. thanks!

[jcohenadad]

Exp_1. Registration using PAM50:

you should not be using -c t1 because this is a t2w like contrast

[jcohenadad]

you should never do straightening with 5mm thick images

[Nilser3]

Thank you @jcohenadad following your suggestion, I worked on the registration of anisotropic images in the subject space:

Registration using AMU7T in subject space:

sct_register_to_template -i image.nii.gz -s image_wm.nii.gz -l landmarks.nii.gz -c t1 -v 1   -t ../../../template_AMU7T/ -s-template-id 4
sct_apply_transfo -i ../../../template_AMU7T/template/AMU7T_wm_qt1.nii.gz  -d image.nii.gz -w  warp_template2anat.nii.gz -o wm_amu7Ttoanat.nii.gz -x nn
sct_register_multimodal -i template2anat.nii.gz -d image.nii.gz -iseg wm_amu7Ttoanat.nii.gz  -dseg image_wm.nii.gz -o amu7Ttoanat_t1q.nii.gz -m sc_dil.nii.gz -param step=1,type=seg,algo=slicereg,metric=MeanSquares:step=2,type=im,algo=syn,metric=CC:step=3,type=seg,algo=syn,metric=CC,gradStep=0.2

Time = 788 sec.

Atlas AMU7T to subject space using 2 warps

sct_apply_transfo -i ../../../template_AMU7T/atlas/AMU7T_50_labels.nii.gz  -d image.nii.gz -w warp_template2anat.nii.gz warp_template2anat2image.nii.gz  -o atlasAMU7T_to_subject.nii.gz -x nn

Register AMU7T atlas using sct_warp_template(in 2 times)

sct_warp_template -d image.nii.gz -w warp_template2anat.nii.gz -t ../../../template_AMU7T/  -ofolder warp_template_1/
sct_warp_template -d image.nii.gz -w warp_template2anat2t1q_aniso_or_cor_den.nii.gz -t warp_template_1/label/ -ofolder warp_template_2

Visualization of external regions (CSF contour, Central-zone, Anterior-fissure and Septum)

In template_AMU7T/atlas/.. the labels are binary, however, after this registration, non-binary masks were obtained, possibly due to linear interpolations when applying the 2 warps.

Metric extraction using sct_extract_metric

sct_extract_metric -i image.nii.gz -f label/atlas/AMU7T_label_353_bin.nii.gz -method wa -o WA_353_GM_R_fasciculus_dorsolateralis.csv

Weighted average metric (WA ± SD) in the non-binary GM Right fasciculus-dorsolateralis mask per slice,in a qT1 image (peak in a MS lesion).

Note:

I worked on a second version of the AMU7T atlas and template (XY zero padding = 256x256) or (45x45mm) to preserve the spinal canal and vertebral bodies on the registered images.

[jcohenadad]

sct_register_to_template -i image.nii.gz -s image_wm.nii.gz -l landmarks.nii.gz -c t1 -v 1   -t ../../../template_AMU7T/ -s-template-id 4

To help the team review your code, I would add a comment that template-id 4 refers to the WM segmentation of the template: https://github.com/spinalcordtoolbox/template_AMU7T/blob/8d68e44d9da9511772d83ffee57b0b094664496f/template/info_label.txt#L8

sct_apply_transfo -i ../../../template_AMU7T/template/AMU7T_wm_qt1.nii.gz  -d image.nii.gz -w  warp_template2anat.nii.gz -o wm_amu7Ttoanat.nii.gz -x nn

Why are you using '-x nn'? And for what purpose are you bringing AMU7T_wm_qt1.nii.gz into the subject's space? (Comments would be welcome)

sct_register_multimodal -i template2anat.nii.gz -d image.nii.gz -iseg wm_amu7Ttoanat.nii.gz  -dseg image_wm.nii.gz -o amu7Ttoanat_t1q.nii.gz -m sc_dil.nii.gz -param step=1,type=seg,algo=slicereg,metric=MeanSquares:step=2,type=im,algo=syn,metric=CC:step=3,type=seg,algo=syn,metric=CC,gradStep=0.2

Why not running these registration parameters while doing sct_register_to_template and specifying a -param flag in the first place?

Sub-sections "Atlas AMU7T to subject space using 2 warps" and "Register AMU7T atlas using sct_warp_template (in 2 times)": what are these? Are they different steps for doing the template to image registration, or are they different strategies that you are comparing? Please describe what is the purpose of each sub-section.

Section "Metric extraction using sct_extract_metric": very cool result!

[vcallot]

Thank you ++ @Nilser3 @jcohenadad and the team for all the efforts for integrating the AMU7T into the SCT!

A few comments and questions :

1) - the native image resolution you are working on is 0.30.34 mm3 (ie. anisotropic, with thickness of 4mm, not 5mm)(but that's no big deal ..)

2) -

following your suggestion, I worked on the registration of anisotropic images in the subject space:

I agree anisotropic images are not adequate for the straightening. In the ISMRM abstract, Arnaud indeed registered the template to the subject image. But then, the warping to the AMU7T template was not adequate (-> could you double check that everything is going well when you go back in the template space if you adopt this "template->subject" strategy?)

Also, how many landmarks do you use when running sct_register_to_template -l landmarks? On the native anisotropic T1q images, the vertebrae can not be seen (not as clearly as with other modalities). Consequently, 2 labels may not be sufficient/adequate.

3) - when you use the option "-c t1", registration is made from (or to) the AMU26_qt1.nii.gz, right?

4) -

I worked on a second version of the AMU7T atlas and template (XY zero padding = 256x256) or (45x45mm) to preserve the spinal canal and vertebral bodies on the registered images.

could you please provide more details? Did you start from scratch with a new crop for the native image? Or something else ?

5) -

Weighted average metric (WA ± SD) in the non-binary GM Right fasciculus-dorsolateralis mask per slice,in a qT1 image (peak in a MS lesion).

Thanks for this preliminary try! When ready with the AMU7T template registration, we will have to investigate more carefully the robustness of the registration when a lesion (or compression, etc..) is present. In this try, the red ROI (supposedly GM fasciculus-dorsolateralis) is more likely a mix of GM and lateral CST. But for a 1st try, this is very cool. Thanks again!

[jcohenadad]

1) - the native image resolution you are working on is 0.30.34 mm3 (ie. anisotropic, with thickness of 4mm, not 5mm)(but that's no big deal ..)

Always good to be precise, thank you. For more transparency and make sure the dev team and @Nilser3 are on the same page with debugging, @Nilser3 can you please upload a 'test' data that will be used as functional tests for the registration commands we are discussing here?

I agree anisotropic images are not adequate for the straightening. In the ISMRM abstract, Arnaud indeed registered the template to the subject image. But then, the warping to the AMU7T template was not adequate (-> could you double check that everything is going well when you go back in the template space if you adopt this "template->subject" strategy?) Also, how many landmarks do you use when running sct_register_to_template -l landmarks? On the native anisotropic T1q images, the vertebrae can not be seen (not as clearly as with other modalities). Consequently, 2 labels may not be sufficient/adequate.

If using more than 2 labels, the algorithm also performs a through-plane non-linear registration, which is to be avoided with thick slices (to minimize interpolation errors). So I would stick with 2 labels and -ref subject instead of -ref template to bypass straightening.

when you use the option "-c t1", registration is made from (or to) the AMU26_qt1.nii.gz, right?

It's a bit convoluted, but "-c t1" uses label_id=0, as per this code, which indeed refers to AMU26_qt1.nii.gz as per this code:

https://github.com/spinalcordtoolbox/template_AMU7T/blob/8d68e44d9da9511772d83ffee57b0b094664496f/template/info_label.txt#L4

could you please provide more details? Did you start from scratch with a new crop for the native image? Or something else ?

I agree more details would be important for the dev team to be able to reproduce the results. This is why a 'test' image is useful, as well as a full script to run the registration. @joshuacwnewton @mguaypaq any suggestions where to upload the data/script?

[joshuacwnewton]

This is why a 'test' image is useful, as well as a full script to run the registration. @joshuacwnewton @mguaypaq any suggestions where to upload the data/script?

While working with the Phillips level files in https://github.com/spinalcordtoolbox/PAM50/pull/3, we used a code/ folder in the root of the repository to store processing scripts as well as test data. Perhaps we could use the same idea here?

e.g. @Nilser3 creates a separate branch on the AMU7T repo, opens a PR for that branch, then makes sure to commit each update to the registration workflow as a change to the processing script.

That way, all someone has to do is clone the repo and check out the branch to get access to the script and test data. (And, any further discussion can be direct PR reviews highlighting specific lines of the script itself.)

[Nilser3]

Thank you @jcohenadad

I have shared a subject HC for test and AMU7T v2 (zero padding in XY plane, dim =256x256) in this PR #6

sct_apply_transfo -i ../../../template_AMU7T/template/AMU7T_wm_qt1.nii.gz  -d image.nii.gz -w  warp_template2anat.nii.gz -o wm_amu7Ttoanat.nii.gz -x nn

Why are you using '-x nn'? And for what purpose are you bringing AMU7T_wm_qt1.nii.gz into the subject's space? (Comments would be welcome)

I consider that the first registration is not enough to have the Template and Subject correctly aligned, so I use the warp of this first coregistration (warp_template2anat) to bring the WM of AMU7T into a "unadjusted subject space", then I add a second coregistration, based on the images and the WM masks between AMU7T in the "unadjusted subject space" and the Subject. Why -x nn? Linear interpolation tends to introduce blurred edges on the contours, so as our original mask is soft,to preserve this degree of softness I chose to use -x nn

sct_register_multimodal -i template2anat.nii.gz -d image.nii.gz -iseg wm_amu7Ttoanat.nii.gz  -dseg image_wm.nii.gz -o amu7Ttoanat_t1q.nii.gz -m sc_dil.nii.gz -param step=1,type=seg,algo=slicereg,metric=MeanSquares:step=2,type=im,algo=syn,metric=CC:step=3,type=seg,algo=syn,metric=CC,gradStep=0.2

Why not running these registration parameters while doing sct_register_to_template and specifying a -param flag in the first place?

Certainly, I made an experience with 5 steps in -param:

sct_register_to_template -i image.nii.gz -s image_wm.nii.gz -l landmarks.nii.gz -c t1 -v 1 -s-template-id 4  -t template_AMU7T  -s-template-id 4 -param step=1,type=imseg,algo=centermassrot,rot_method=pcahog:step=2,type=seg,algo=bsplinesyn,slicewise=0,metric=MeanSquares,samplStrategy=None,samplPercent=0.2,iter=3,smooth=1:step=3,type=seg,algo=slicereg,metric=MeanSquares:step=4,type=im,algo=syn,metric=CC:step=5,type=seg,algo=syn,metric=CC,gradStep=0.2

the first 2 (the default steps from sct_register_to_template) + 3 steps used with sct_register_multimodal, The results were not the same because in our sct_register_multimodal stage, we used a dilated SC mask (-m sc_dil) to improve the accuracy in the region of interest, so I will continue to optimize the params so that in a single sct_register_to_template we have a more accurate registration.

Sub-sections "Atlas AMU7T to subject space using 2 warps" and "Register AMU7T atlas using sct_warp_template (in 2 times)": what are these? Are they different steps for doing the template to image registration, or are they different strategies that you are comparing? Please describe what is the purpose of each sub-section.

There are two steps I use to register the AMU7T atlas into the subject space, the first step register the AMU7T atlas into the "unadjusted subject space" and the second step takes this unadjusted atlas and register it into the Subject space.
I know it is a redundant process and creates a unnecessary atlases, it will be simplified if we have only one Warp.

[Nilser3]

Hi @vcallot

I worked on a second version of the AMU7T atlas and template (XY zero padding = 256x256) or (45x45mm) to preserve the spinal canal and vertebral bodies on the registered images.

could you please provide more details? Did you start from scratch with a new crop for the native image? Or something else ?

I made a zero paddingaround the XY plane without changing neither the PAM50 reference, nor the resolution, nor any interpolation, here is this version, however I have an issue about it: In our original template AMU72_t2s: the intensities starts with negative values, here the original histogram: here the histogram with zero padding: the zero padding affects this histogram, then maybe we should make a histogram shift, so that it starts from ZERO in the original image? before making the padding, so that ZERO is the background.

This problem is not seen in the qT1 template (fortunately). Original histogram:

Zero padding histogram:

5) -

Weighted average metric (WA ± SD) in the non-binary GM Right fasciculus-dorsolateralis mask per slice,in a qT1 image (peak in a MS lesion).

Thanks for this preliminary try! When ready with the AMU7T template registration, we will have to investigate more carefully the robustness of the registration when a lesion (or compression, etc..) is present. In this try, the red ROI (supposedly GM fasciculus-dorsolateralis) is more likely a mix of GM and lateral CST. But for a 1st try, this is very cool. Thanks again!

of course 👍

[jcohenadad]

From https://github.com/spinalcordtoolbox/template_AMU7T/issues/4#issuecomment-1641526025:

step=3,type=seg,algo=slicereg,metric=MeanSquares

This step is irrelevant bc at step1 you already do a centermass rot

step=4,type=im,algo=syn,metric=CC:step=5,type=seg,algo=syn,metric=CC,gradStep=0.2

It is unusual to precede seg-based with image-based

[Nilser3]

Hi @jcohenadad @vcallot

Here is an improved registration based on the following command lines :

sct_register_to_template -i image.nii.gz -s image_wm.nii.gz -l landmarks.nii.gz -c t1 -v 1 -s-template-id 4  -t ../template_AMU7T  -param step=1,type=imseg,algo=centermassrot,rot_method=pcahog:step=2,type=seg,algo=bsplinesyn,slicewise=0,metric=MeanSquares,samplStrategy=None,samplPercent=0.2,iter=2,smooth=1,rot_method=pcahog:step=3,type=seg,algo=syn,metric=MeanSquares,shrink=2,dof=Tz_Rz_Sz,slicewise=1,iter=20 -ref subject

sct_apply_transfo -i ../template_AMU7T/atlas/AMU7T_50_labels.nii.gz -d image.nii.gz -w warp_template2anat.nii.gz -o atlas_2_subject.nii.gz -x nn

sct_warp_template -d image.nii.gz -w warp_template2anat.nii.gz  -t ../template_AMU7T

previously I had to make a manual correction to the WM mask (red outline in the video), and if we compare it with the registered AMU7T probabilistic WM mask via sct_warp_template (light blue mask in the video) a better correspondence of masks is observed.

Video

amu7t.webm

The registered atlas presents some aberrant deformations (43 - 50 sec in the video), which were excluded for the quantification presented in Fig. 01 (notebook here)

Fig. 01

[jcohenadad]

That looks pretty good @Nilser3, good job! 👏

[vcallot]

Cool! Great job @Nilser3 ! Looks nice and hope it behave the same with different subjects!