Visible outliers for T1w and T2w atrophy results

[PaulBautin]

Results on all subjects for T1w images make much more sense! However, there are still evident outliers (mostly overestimated CSAs). Inspection of these outliers often show visible motion, poor definition and different image resolution (than the generic resolutions). In addition about 38 (t1w) and 5 (t2w) vertebrae levels have been removed because they do not present any CSA measurements.

Surprisingly, it is the same subjects in t1w and t2w images that are evident outliers. Should I try to add a criteria to remove outliers as they are often image related and do not reflect reality?

[jcohenadad]

Results on all subjects for T1w images make much more sense!

awesome! could you please indicate the commit number that you used to generate those results?

However, there are still evident outliers (mostly overestimated CSAs). Inspection of these outliers often show visible motion, poor definition

so i guess results make sense, but i would be curious to look at native and resampled segmentations. Could you upload an example?

and different image resolution (than the generic resolutions).

don't you resample everything to 1mm at the beginning?

In addition about 38 (t1w) and 5 (t2w) vertebrae levels have been removed because they do not present any CSA measurements.

what do you mean by "vertebrae levels were removed"? could you point to the code that does that?

Surprisingly, it is the same subjects in t1w and t2w images that are evident outliers. Should I try to add a criteria to remove outliers as they are often image related and do not reflect reality?

not sure-- at this point i'd like to understand better what's going on

[PaulBautin]

Results ^ were run with commit: https://github.com/sct-pipeline/csa-atrophy/pull/82/commits/9a343c3021fbae54dce12631c2bf041623f520f4 Data: https://github.com/spine-generic/data-multi-subject/releases/tag/r20201001

don't you resample everything to 1mm at the beginning?

Indeed, if native images do not have generic resolution they are resampled. However, there seems to be a prevalence of errors in these resampled images. (i generalized this from a very small sample it has to be proven)

Could you upload an example?

Native image resampled to 1x1x1mm: sub-brnoUhb03_T1w_RPI_r_crop.zip sub-brnoUhb03_T1w_RPI_r_seg.zip

Image resampled and transformed (without change of affine array): sub-brnoUhb03_T1w_RPI_r_crop_r0.93_t1.zip sub-brnoUhb03_T1w_RPI_r_crop_r0.93_t1_seg.zip

what do you mean by "vertebrae levels were removed"? could you point to the code that does that?

It happens that CSA is not computed for certain vertebrae levels. In that case values are removed from dataframe. Line 285 of https://github.com/sct-pipeline/csa-atrophy/blob/master/csa_rescale_stat.py:

# identify rows with missing values
print("Remove rows with missing values...")
lines_to_drop = df_vert[df_vert['MEAN(area)'] == 'None'].index
df_vert = df_vert.drop(df_vert.index[lines_to_drop])
df_vert['MEAN(area)'] = pd.to_numeric(df_vert['MEAN(area)'])
print("  Rows removed: {}".format(lines_to_drop))

[jcohenadad]

Native image resampled to 1x1x1mm: sub-brnoUhb03_T1w_RPI_r_crop.zip

this imaged is cropped-- not only resampled. Maybe the problem comes from there? vertebral labeling won't work well on this cropped image

[PaulBautin]

this imaged is cropped-- not only resampled

Vertebrae labeling is performed on non-cropped image (file is to big to put here). Vertebrae labeling results are: sub-brnoUhb03_T1w_RPI_r_seg_labeled.zip Then cropped: sub-brnoUhb03_T1w_RPI_r_seg_labeled_crop.zip

From investigation, high intra-subject variability (inter-vertebrae + inter-transformation) correlates well with the subject being an oulier. This is most probably due to poor image quality causing segmentation abnormalities.

[jcohenadad]

I think the reason for the overestimation of the CSA with the increased rescaling, is caused by an increasing partial volume effect with tissue outside of the CSF (which has similar intensity as the cord). The fact that the T1w are lower res than the T2w data from the start (1mm vs. 0.8mm), this does not help. It would be interesting to do the same rescaling experiment, but resampling the T2w data to 1mm at the very beginning, to see if results are closer to the T1w data, which would confirm this hypothesis.

As for the outliers reported in https://github.com/sct-pipeline/csa-atrophy/issues/83#issue-722482133, they could very well be associated with subjects with more motion (more blurriness), and as rescaling goes up, the partial volume effect would be worse. The problem is that these "bad" subjects give a biased view of the rescaling experiment. I am wondering how results would look like if you only selected "good" subjects (ie without visible motion).

[PaulBautin]

I tried resampling images t2w images to 1x1x1mm to see if results were closer to those obtained with the t1w data. It is not obvious, there still is an evident over-estimation of CSA after rescaling, i wonder why this time around outliers seem to be better distributed around each boxplot. Note: inter-subject variability increases with larger resampling

[jcohenadad]

very interesting. this result should probably go into an annex of your manuscript. i’m still unsure how to interpret this.

[jcohenadad]

@PaulBautin i'd like to dig into the results you presented here: https://github.com/sct-pipeline/csa-atrophy/issues/83#issuecomment-711417407, but first I want to make sure i'm looking at the exact same code as the one you ran. Can you please add the commit number + syntax + config files

[PaulBautin]

Results for T2w resampled to 0.8x0.8x0.8mm: commit: https://github.com/sct-pipeline/csa-atrophy/pull/82/commits/744ca79f1320ecf90d0aa8c8ac12f7ff40fe7770

Results for T2w resampled to 1x1x1mm: commit: https://github.com/sct-pipeline/csa-atrophy/pull/85/commits/0322ce67982d6b4530ecddd6b4245705162de996

As you pointed out in comment: https://github.com/sct-pipeline/csa-atrophy/pull/85#discussion_r506773120, I have been setting all "local" configs directly in the GitHub directory, so the config in the commits is the one i used.

For syntax, on compute canada i have been using: run_all -config config_sct_run_batch.yml -job-template job_template.sh -n 32 and locally: sct_run_batch -config config_sct_run_batch.yml

[PaulBautin]

@jcohenadad. The merge of PR #89, has not modified the results and the number of outliers. Used repo for t1: https://github.com/PaulBautin/csa-atrophy/tree/compute_canada Used repo for t2: https://github.com/sct-pipeline/csa-atrophy

[jcohenadad]

@PaulBautin would it be possible to have a side-by-side comparison of T1w results before/after the PR? I would expect to see differences, since visually it was clear that some segmentations were cropped with the "before" approach

[PaulBautin]

With PR #89 the outliers seem to be more aligned but nothing obvious:

repo for results before #89: https://github.com/sct-pipeline/csa-atrophy/tree/stat_cov_err repo for results after #89: https://github.com/PaulBautin/csa-atrophy/tree/compute_canada

[jcohenadad]

nothing obvious indeed, but results seem less spread out (less outliers, smaller quartiles), which is good.