Open valosekj opened 1 year ago
I can reproduce this issue using testing data, too:
master|joshua@tadpole:~/repos/spinalcordtoolbox$ cd data/sct_example_data/t1
master|joshua@tadpole:~/repos/spinalcordtoolbox/data/sct_example_data/t1$ sct_deepseg_sc -i t1.nii.gz -c t1
--
Spinal Cord Toolbox (git-master-9f4d05ad2ce9bd48fd5a396310597fb5db645253)
sct_deepseg_sc -i t1.nii.gz -c t1
--
[...]
Done! To view results, type:
fsleyes /home/joshua/repos/spinalcordtoolbox/data/sct_example_data/t1/t1.nii.gz -cm greyscale /home/joshua/repos/spinalcordtoolbox/data/sct_example_data/t1/t1_seg.nii.gz -cm red -a 70.0 &
master|joshua@tadpole:~/repos/spinalcordtoolbox/data/sct_example_data/t1$ sct_image -i t1.nii.gz -header | grep dim
dim [3, 192, 260, 320, 1, 0, 0, 0]
pixdim [1.0, 0.999996, 1.0, 1.0, 2.0, 0.0, 0.0, 0.0]
phase_dim 1
freq_dim 2
slice_dim 3
master|joshua@tadpole:~/repos/spinalcordtoolbox/data/sct_example_data/t1$ sct_image -i t1_seg.nii.gz -header | grep dim
dim [3, 192, 260, 320, 1, 1, 1, 1]
pixdim [1.0, 0.999996, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
phase_dim 0
freq_dim 0
slice_dim 0
I'll dig into the cause next.
This behavior comes from nibabel
:
dim
and pixdim
fields have 8 values.pixdim[0]
is reserved for metadata purposes.dim
/pixdim[1:3]
, 4D images use dim
/pixdim[1:4]
, etc. for 5D, 6D, up to 7D.dim
/pixdim
that AREN'T used (depending on image size) seem to have no defined purpose.dim
/pixdim
values. dim
/pixdim
values, therefore it sets them to 1.0
.See also:
The most relevant quote from that issue is:
The standard provides no interpretation for dimensions beyond
dim[0]
or guidance on what they should be, so there is no correct/incorrect setting. They should not be considered to have any impact on the interpretation of the data at all.
Your input image is 3D, meaning that only the values [0:3]
should be relevant; [4:7]
should be "unused". (i.e. set to 0
, 1
, whatever...)
So, my question is: Why does the nnU-Net
software care about the "unused" dim/pixdim fields? This seems overly sensitive to me.
To fix this on SCT's end, we would have to store the original values of dim[4:7]
and pixdim[4:7]
ourselves in our Image
objects. (But, I'm not sure if nibabel
even provides access to the other values?)
Unfortunately, even if we are able to access the old dim
/pixdim
values, we are blocked by another issue: https://github.com/spinalcordtoolbox/spinalcordtoolbox/issues/2462#issuecomment-1396266576. Right now, SCT's Image
objects mishandle dim[4:7]
and pixdim[4:7]
, so we would have to rewrite that part of Image
, which is a pretty significant effort. :sweat:
Thanks for your swift digging into this issue, @joshuacwnewton!
So, my question is: Why does the
nnU-Net
software care about the "unused" dim/pixdim fields? This seems overly sensitive to me. Especially given nipy/nibabel#1034 (comment):The standard provides no interpretation for dimensions beyond
dim[0]
or guidance on what they should be, so there is no correct/incorrect setting. They should not be considered to have any impact on the interpretation of the data at all.
This is what I am also wondering! I will explore the nnU-Net
more deeply to find out the reason!
After deeper digging, it seems that the nnUNet errors are not caused by pixdim
how I initially thought:
Notice the difference
2.5, 0.889114, 0.0, 0.0, 0.0
vs2.499999, 1.0, 1.0, 1.0, 1.0
. This difference is consequently raising errors (here and here) when usingnnU-Net
package.
It also seems that the problem is not related to nibabel
but to SimpleITK.
Thanks to my colleague @KaterinaKrejci231054 for help with debugging!
So, to summarize this issue, there are actually 2 separate differences between the anat/seg images:
dim
/pixdim
values. (nibabel
)direction
matrix. (SimpleITK
)However, only 2. is what causes the error in nnU-Net
. (In other words, 1. was a red herring.)
I see! Great investigation! :tada:
EDIT: See #4025 for the conclusion.
I have some follow-up questions:
Good questions! I don't have the answers right now. We will continue with further digging.
Might be remotely related to https://github.com/ivadomed/model_seg_sci/issues/21. It seems that torchIO
also uses SimpleITK
.
Just for the record, I encountered the same ITK direction error also for sci-zurich
dataset after using sct_crop_image
and sct_resample
(used preprocessing script available here).
Again, I fixed the issue using change_itk_direction.py:
for sub in sub*;do cd $sub; for ses in ses*;do cd $ses/anat; python ~/code/change_itk_direction.py ${sub}_${ses}_acq-sag_T2w.nii.gz ../../../derivatives/labels/${sub}/${ses}/anat/${sub}_${ses}_acq-sag_T2w_lesion-manual.nii.gz; cd ../..; done; cd ..;echo $sub done;done
The script raised an error for sub-zh21
, though:
Traceback (most recent call last):
File "/home/GRAMES.POLYMTL.CA/p118175/code/change_itk_direction.py", line 15, in <module>
image1 = sitk.ReadImage(sys.argv[1])
File "/home/GRAMES.POLYMTL.CA/p118175/code/model_seg_sci/venv/lib/python3.10/site-packages/SimpleITK/extra.py", line 355, in ReadImage
return reader.Execute()
File "/home/GRAMES.POLYMTL.CA/p118175/code/model_seg_sci/venv/lib/python3.10/site-packages/SimpleITK/SimpleITK.py", line 8438, in Execute
return _SimpleITK.ImageFileReader_Execute(self)
RuntimeError: Exception thrown in SimpleITK ImageFileReader_Execute: /tmp/SimpleITK-build/ITK/Modules/IO/NIFTI/src/itkNiftiImageIO.cxx:2016:
ITK ERROR: ITK only supports orthonormal direction cosines. No orthonormal definition found!
Thus, I removed the sub-zh21
subject entirely from the training dataset. Now nnUNetv2_plan_and_preprocess --verify_dataset_integrity
is happy.
Tagging @naga-karthik to let him know.
Hi all,
I came across similar problems with discrepancies in direction matrices throwing errors when training with torchio, and also solved by using the method .SetDirection(). I dug a bit into what might cause it on SITK side and so here's my take on your questions above. I'm no expert in SITK nor C++, so take with a grain of salt :)
* What does the `direction` matrix represent? And, how is it computed? (In other words, how is it different than the various NIfTI1 image properties such as qform/sform matrices?)
It seems the direction matrix will be taken from either the qform or sform matrix based on the logic here. If sform and qform are "sufficiently" similar, then sform is used for direction matrix. Otherwise, if sform is orthonormal and sform_code==NIFTI_XFORM_SCANNER_ANAT, again sform is used. And then there are several other conditions to decide whether to use sform or qform.
Another difference is that the spacing/zooms are incorporated into the sform/qform matrices but seem not to be taken into account for the SITK direction matrix.
* What is causing the differences in the `direction` matrix between anat/seg images?
My guess is that it's because the anat image has sform_code==NIFTI_XFORM_SCANNER_ANAT and so sform will be used to get the direction matrix whereas the seg image has sform_code==NIFTI_XFORM_ALIGNED_ANAT and so the qform may be used. Besides these differences, nearly every other piece of metadata is the same in both images.
im_anat = sitk.ReadImage('sub-001_T1w.nii.gz')
im_seg = sitk.ReadImage('sub-001_T1w_seg.nii.gz')
# Print out all metadata for both images side by side
for key in im_anat.GetMetaDataKeys():
print(f'{key}: Anat: {im_anat.GetMetaData(key)}\t\tSeg: {im_seg.GetMetaData(key)}')
Thank you so much for your insight!
If I'm understanding correctly, since SimpleITK's getDirection()
fetches the anat image's sform
(1) and the seg image's qform
(2)...
If the anat image's sform
(1):
> fslhd sub-001_T1w.nii.gz
sform_name Scanner Anat
sform_code 1
sto_xyz:1 -0.351562 0.000000 -0.000166 90.746681
sto_xyz:2 -0.000007 0.334166 0.776687 -103.529266
sto_xyz:3 -0.000022 -0.109222 2.376290 -57.177719
sto_xyz:4 0.000000 0.000000 0.000000 1.000000
Matched the seg image's qform
(2):
> fslhd sub-001_T1w_seg.nii.gz
qform_name Aligned Anat
qform_code 2
qto_xyz:1 -0.351562 -0.000004 -0.000004 90.746681
qto_xyz:2 -0.000004 0.334166 0.776687 -103.529266
qto_xyz:3 0.000001 -0.109222 2.376290 -57.177719
qto_xyz:4 0.000000 0.000000 0.000000 1.000000
Then there would be no discrepancy between the two "direction matrices", and thus no error?
If this is the case, it seems that this is yet another issue due to discrepancies in between the qform and sform, and we've come across many:
Thankfully though, it seems that this issue doesn't stem from SCT, at least (because sct_deepseg_sc doesn't modify the sform/qform -- they are mismatched even within just the input anat image).
Actually, maybe sct_deepseg_sc
is at fault here: If wasn't changing the codes of the segmented image from 1
(Scanner anat) to 2
(Aligned anat), then the sform
would be fetched both times, avoiding the error. (Since the segmentation is in the same space as the anat image, shouldn't the codes be preserved?)
Just to clean up this issue a little (since the title of this issue focuses on the dim/pixdim 1.0
issue), I've summarized the conclusions of the SITK-focused investigations in a separate issue:
And I've hidden the SITK-specific discussion as "off-topic", just so that we keep the discussion to one problem per issue.
Again, thank you so much @rickymwalsh! :hearts:
Description
When running:
The output SC seg has a different
pixdim
that the input image:Notice the difference
2.5, 0.889114, 0.0, 0.0, 0.0
vs2.499999, 1.0, 1.0, 1.0, 1.0
.~This difference is consequently raising errors (here and here) when using
nnU-Net
package.~ Update: See https://github.com/spinalcordtoolbox/spinalcordtoolbox/issues/4025#issuecomment-1445193122.Example data available here.