Proposals for ANTs-based solution

[stnava]

This may provide a better starting point for your parameter tuning:

1. make proper masks via ThresholdImage

ThresholdImage 3 Sub1_moving.nii.gz Sub1_mask.nii.gz 0.01 Inf
ThresholdImage 3 Sub2_moving.nii.gz Sub2_mask.nii.gz 0.01 Inf
antsRegistrationSyNQuick.sh -d 3 -f Sub1_moving.nii.gz -m fixed_image.nii.gz -t r -x Sub1_mask.nii.gz -o /tmp/A1 
antsRegistrationSyNQuick.sh -d 3 -f Sub2_moving.nii.gz -m fixed_image.nii.gz -t r -x Sub2_mask.nii.gz -o /tmp/A2

One should also try making a mask estimate for the fixed image, if possible. Is this intra-subject data? Hard to tell. Usually we are solving these problem with intra-subject data whereas, here, it seems you have raw data and then some form of average.

2. Downsample to speed up experiments

ResampleImageBySpacing 3 Sub2_moving.nii.gz Sub2_moving2.nii.gz 2 2 2 1

2. use antsAI ( to search parameter space ) or invariantImageSimilarity in ANTsR

These will allow you to search along a relevant parameter space. Example:

library( ANTsR )
 fi<-antsImageRead( 'fixed_image2.nii.gz' )
 mi<-antsImageRead( 'Sub2_moving2.nii.gz' )
msk = getMask( mi )
thSearch = seq( -90, 90, by = 10 )
mival<-invariantImageSimilarity( mi, fi, 
     localSearchIterations = 0,   # increase this later once you get an idea of what parameters 'work'
     thetas3 = thSearch, 
     thetas = thSearch, 
     thetas2 = thSearch, transform = 'Rigid', mask = msk )
mapped = antsApplyTransforms( mi, fi, transformlist=mival[[2]], whichtoinvert=FALSE )
plot( mi,  mapped, axis=1, alpha=0.5 )
# refine with a call to antsregistration
 fi<-antsImageRead( 'fixed_image.nii.gz' )
 mi<-antsImageRead( 'Sub2_moving.nii.gz' )
areg = antsRegistration( mi, fi, typeofTransform="Rigid", mask = msk,
       initialTransform=mival[[2]] )

anyway - these are my current thoughts.

[archerdb]

Thanks for the detailed help! I'll work on this with my data.

Thanks again, Derek