Closed andrew-yian-sun closed 6 months ago
seems to be specifically nibabel
issue, and you are using older nibabel (0.10.0). In the newer version there is newer nibabel
yoh@typhon:~$ time docker run --rm --entrypoint python nipy/heudiconv:0.10.0 -c 'import nibabel; print(nibabel.__version__)'
3.2.2
yoh@typhon:~$ time docker run --rm --entrypoint python nipy/heudiconv:latest -c 'import nibabel; print(nibabel.__version__)'
5.0.1
so can you try with nipy/heudiconv:0.12.2
and not the outdated :latest
?
Thanks for the suggestion - however I seemed to have run into the same error with 0.12.2 as well:
> time docker run --rm \
> --volume /Volumes/LCproject/LC_aging/data_mri:/base \
> nipy/heudiconv:0.12.2 \
> --dicom_dir_template /base/data_BAP/{subject}_Session*{session}_*/*/* \
> --outdir /base/data_BAP_BIDS \
> --heuristic /base/data_BAP_BIDS/code/heuristic_postUpdate.py \
> --subjects BAP120 \
> --ses vis \
> --converter dcm2niix \
> --bids \
> --overwrite
INFO: Running heudiconv version 0.12.2 latest 0.12.2
INFO: Need to process 1 study sessions
INFO: PROCESSING STARTS: {'subject': 'BAP120', 'outdir': '/base/data_BAP_BIDS/', 'session': 'vis'}
INFO: Processing 1370 dicoms
INFO: Analyzing 1370 dicoms
/opt/miniconda-py39_4.12.0/lib/python3.9/site-packages/nibabel/nicom/dicomwrappers.py:525: UserWarning: Derived images found and removed
warnings.warn('Derived images found and removed')
Traceback (most recent call last):
File "/opt/miniconda-py39_4.12.0/bin/heudiconv", line 8, in <module>
sys.exit(main())
File "/src/heudiconv/heudiconv/cli/run.py", line 24, in main
workflow(**kwargs)
File "/src/heudiconv/heudiconv/main.py", line 379, in workflow
prep_conversion(sid,
File "/src/heudiconv/heudiconv/convert.py", line 173, in prep_conversion
seqinfo = group_dicoms_into_seqinfos(
File "/src/heudiconv/heudiconv/dicoms.py", line 202, in group_dicoms_into_seqinfos
mwinfo = validate_dicom(filename, dcmfilter)
File "/src/heudiconv/heudiconv/dicoms.py", line 106, in validate_dicom
del mw.series_signature[sig]
File "/opt/miniconda-py39_4.12.0/lib/python3.9/site-packages/nibabel/onetime.py", line 142, in __get__
val = self.getter(obj)
File "/opt/miniconda-py39_4.12.0/lib/python3.9/site-packages/nibabel/nicom/dicomwrappers.py", line 625, in series_signature
signature['image_shape'] = (self.image_shape, eq)
File "/opt/miniconda-py39_4.12.0/lib/python3.9/site-packages/nibabel/onetime.py", line 142, in __get__
val = self.getter(obj)
File "/opt/miniconda-py39_4.12.0/lib/python3.9/site-packages/nibabel/nicom/dicomwrappers.py", line 545, in image_shape
frame_indices = np.delete(frame_indices, stackid_dim_idx, axis=1)
File "<__array_function__ internals>", line 200, in delete
File "/opt/miniconda-py39_4.12.0/lib/python3.9/site-packages/numpy/lib/function_base.py", line 5136, in delete
axis = normalize_axis_index(axis, ndim)
numpy.AxisError: axis 1 is out of bounds for array of dimension 1
is there a chance you could share that DICOM publicly? privately? I think we should file the issue with nibabel -- I don't see a similar issue there.
@yarikoptic I get the same error as above using version 0.13.1
running in Singularity. All of the DICOMs are Siemens Enhanced DICOMS from the XA20 software version. dcm2niix
behaves fine but nibabel
doesn't like parsing the data:
singularity run --containall -B $PWD:/data /groups/adamraikes/singularity_images/heudiconv_0.13.1.sif -d /data/dicoms/{subject}/*/* -o /data/nifti/ -f convertall -s 000052 -c none
INFO: Running heudiconv version 0.13.1 latest 0.13.1
INFO: Need to process 1 study sessions
INFO: PROCESSING STARTS: {'subject': '000052', 'outdir': '/data/nifti/', 'session': None}
INFO: Processing 1187 dicoms
INFO: Analyzing 1187 dicoms
/opt/miniconda-py39_4.12.0/lib/python3.9/site-packages/nibabel/nicom/dicomwrappers.py:525: UserWarning: Derived images found and removed
warnings.warn('Derived images found and removed')
Traceback (most recent call last):
File "/opt/miniconda-py39_4.12.0/bin/heudiconv", line 8, in <module>
sys.exit(main())
File "/src/heudiconv/heudiconv/cli/run.py", line 30, in main
workflow(**kwargs)
File "/src/heudiconv/heudiconv/main.py", line 463, in workflow
prep_conversion(
File "/src/heudiconv/heudiconv/convert.py", line 216, in prep_conversion
seqinfo = group_dicoms_into_seqinfos(
File "/src/heudiconv/heudiconv/dicoms.py", line 283, in group_dicoms_into_seqinfos
mwinfo = validate_dicom(filename, dcmfilter)
File "/src/heudiconv/heudiconv/dicoms.py", line 125, in validate_dicom
del mw.series_signature[sig]
File "/opt/miniconda-py39_4.12.0/lib/python3.9/site-packages/nibabel/onetime.py", line 156, in __get__
val = self.getter(obj)
File "/opt/miniconda-py39_4.12.0/lib/python3.9/site-packages/nibabel/nicom/dicomwrappers.py", line 625, in series_signature
signature['image_shape'] = (self.image_shape, eq)
File "/opt/miniconda-py39_4.12.0/lib/python3.9/site-packages/nibabel/onetime.py", line 156, in __get__
val = self.getter(obj)
File "/opt/miniconda-py39_4.12.0/lib/python3.9/site-packages/nibabel/nicom/dicomwrappers.py", line 545, in image_shape
frame_indices = np.delete(frame_indices, stackid_dim_idx, axis=1)
File "<__array_function__ internals>", line 200, in delete
File "/opt/miniconda-py39_4.12.0/lib/python3.9/site-packages/numpy/lib/function_base.py", line 5136, in delete
axis = normalize_axis_index(axis, ndim)
numpy.AxisError: axis 1 is out of bounds for array of dimension 1
Note: The containall flag was used to rule out a conflict with nibabel installed locally. There were no differences whether running with --cleanenv
, --containall
, or no flag
Can confirm I'm getting this error as well. Same versions. It is happening on derived TRACEW images from DWI sequences.
Filed https://github.com/nipy/nibabel/issues/1245 since I think the issue is to fix within nibabel. Do you have a sample DICOM you could potentially share? @kodiweera - do we have any TRACEW images on phantoms?
Not any that I'm aware of. However, I think you can derive/calculate the tracew images from dwi data. TRACEW sounds like weighting of the traces of the DTI matrix.
I'm able to send one of ours as an example. As long as its not posted to be publicly accessible. Please advise where I can send.
The problem is caused in this block of heudiconv/dicoms.py
in MultiframeWrapper::image_shape()
at line ~512
if hasattr(first_frame, 'get') and first_frame.get([0x18, 0x9117]):
# DWI image may include derived isotropic, ADC or trace volume
try:
self.frames = pydicom.Sequence(
frame
for frame in self.frames
if frame.MRDiffusionSequence[0].DiffusionDirectionality != 'ISOTROPIC'
)
except IndexError:
# Sequence tag is found but missing items!
raise WrapperError('Diffusion file missing information')
except AttributeError:
# DiffusionDirectionality tag is not required
pass
else:
if n_frames != len(self.frames):
warnings.warn('Derived images found and removed')
n_frames = len(self.frames)
has_derived = True
For a trace image it detects all frames as derived and removes them. You are left with an empty image.
eh, you really confused me with the statement of it being in heudiconv/dicoms.py
since it is not there but in nibabel
❯ pwd
/home/yoh/proj/nipy/nipy-suite/nibabel
❯ git grep -n -C 3 'found and removed'
nibabel/nicom/dicomwrappers.py-522- pass
nibabel/nicom/dicomwrappers.py-523- else:
nibabel/nicom/dicomwrappers.py-524- if n_frames != len(self.frames):
nibabel/nicom/dicomwrappers.py:525: warnings.warn('Derived images found and removed')
nibabel/nicom/dicomwrappers.py-526- n_frames = len(self.frames)
nibabel/nicom/dicomwrappers.py-527- has_derived = True
nibabel/nicom/dicomwrappers.py-528-
--
and that is why should be fixed there, thus https://github.com/nipy/nibabel/issues/1245 . @kodiweera - could we collect some on phantom in coming days? (@chrisadamsonmcri we better have a public copy of it) or may be @neurolabusc has one of those tracew files somewhere among qa_ datasets?
Sorry my bad.
Sample tracew DICOMs were collected and available now at http://datasets.datalad.org/?dir=/dicoms/dartmouth-phantoms/Siemens-DWITrace-20230803 . But upon quick try with heudiconv
- I did not reproduce this issue, so must be something special about original DICOMs.
You may want to have dcm2niix ignore derived images (-i y
). One should expect better derived diffusion measures (TRACE, MD, ADC, FA, etc) after processing with dwidenoise, mrdegibbs, topup and eddy. In general, BIDS treats derivatives differently than source data.
If you do ask dcm2niix to retain derived data, it will add a hint to the JSON indicating that the image has been detected as derived:
"RawImage": false,
However, dcm2niix's derived image detection errs on the side of caution, and therefore not all derived data will be identified as such. Indeed, in many cases whether data is derived or not is in the eye of the beholder, e.g. some users only export images with NonlinearGradientCorrection and use it as their raw data, while other users export with and without this manipulation and likely consider the corrected image derived from the uncorrected image.
You may want to have dcm2niix ignore derived images (
-i y
).
this logic happens in heudiconv before invoking dcm2niix -- we also try to sort dicoms into separate series which we then eventually give to dcm2niix
.
I'm able to send one of ours as an example. As long as its not posted to be publicly accessible. Please advise where I can send.
Since my attempts to reproduce on locally collected data failed, if you could share with me (yarikoptic at gmail) some way -- would be great. I will then look into fixing up nibabel for it.
nevermind! I have gone through all dcm_qa
repos of @neurolabusc and got one
dcm_qa_xa30/In/20_DWI_dir80_AP
INFO: Running heudiconv version 0.13.1.post7+g83c8424.d20230720 latest 0.13.1
INFO: Analyzing 1 dicoms
/home/yoh/proj/heudiconv/heudiconv-master/venvs/dev3/lib/python3.11/site-packages/nibabel/nicom/dicomwrappers.py:525: UserWarning: Derived images found and removed
warnings.warn('Derived images found and removed')
Traceback (most recent call last):
File "/home/yoh/proj/heudiconv/heudiconv-master/venvs/dev3/bin/heudiconv", line 8, in <module>
sys.exit(main())
^^^^^^
File "/home/yoh/proj/heudiconv/heudiconv-master/heudiconv/cli/run.py", line 30, in main
workflow(**kwargs)
File "/home/yoh/proj/heudiconv/heudiconv-master/heudiconv/main.py", line 394, in workflow
study_sessions = get_study_sessions(
^^^^^^^^^^^^^^^^^^^
File "/home/yoh/proj/heudiconv/heudiconv-master/heudiconv/parser.py", line 221, in get_study_sessions
seqinfo_dict = group_dicoms_into_seqinfos(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yoh/proj/heudiconv/heudiconv-master/heudiconv/dicoms.py", line 283, in group_dicoms_into_seqinfos
mwinfo = validate_dicom(filename, dcmfilter)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yoh/proj/heudiconv/heudiconv-master/heudiconv/dicoms.py", line 125, in validate_dicom
del mw.series_signature[sig]
^^^^^^^^^^^^^^^^^^^
File "/home/yoh/proj/heudiconv/heudiconv-master/venvs/dev3/lib/python3.11/site-packages/nibabel/onetime.py", line 156, in __get__
val = self.getter(obj)
^^^^^^^^^^^^^^^^
File "/home/yoh/proj/heudiconv/heudiconv-master/venvs/dev3/lib/python3.11/site-packages/nibabel/nicom/dicomwrappers.py", line 625, in series_signature
signature['image_shape'] = (self.image_shape, eq)
^^^^^^^^^^^^^^^^
File "/home/yoh/proj/heudiconv/heudiconv-master/venvs/dev3/lib/python3.11/site-packages/nibabel/onetime.py", line 156, in __get__
val = self.getter(obj)
^^^^^^^^^^^^^^^^
File "/home/yoh/proj/heudiconv/heudiconv-master/venvs/dev3/lib/python3.11/site-packages/nibabel/nicom/dicomwrappers.py", line 545, in image_shape
frame_indices = np.delete(frame_indices, stackid_dim_idx, axis=1)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yoh/proj/heudiconv/heudiconv-master/venvs/dev3/lib/python3.11/site-packages/numpy/lib/function_base.py", line 5254, in delete
axis = normalize_axis_index(axis, ndim)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
numpy.exceptions.AxisError: axis 1 is out of bounds for array of dimension 1
!
edit: full log with some other errors observed is at http://www.oneukrainian.com/tmp/dcm_qa_heudiconv_sweep.txt
@chrisadamsonmcri I think you nailed the reason in https://github.com/nipy/heudiconv/issues/670#issuecomment-1657327970 ! FWIW, if I disable that removal of isotropic images, and use "experimental" dcm2niix and check what dcm2niix assigns for BidsGuess, I get:
❯ rm -rf /tmp/out; PATH=/home/yoh/deb/gits/pkg-exppsy/dcm2niix/build/bin/:$PATH HEUDICONV_LOG_LEVEL=ERROR heudiconv --dbg -f convertall --bids -o /tmp/out --files dcm_qa_xa30/In/20_DWI_dir80_AP -s NA -ss ses1 -l ''; cat /tmp/out/.heudiconv/NA/ses-ses1/info/NA_ses-ses1.auto.txt; ls -l /tmp/out; grep -A3 Guess /tmp/out/*json
INFO: Running heudiconv version 0.13.1.post7+g83c8424.d20230720 latest 0.13.1
...
INFO: PROCESSING DONE: {'subject': 'NA', 'outdir': '/tmp/out/', 'session': 'ses1'}
{('run{item:03d}', ('nii.gz',), None): ['20-DWI_dir80_AP']}total 140
-rw------- 1 yoh yoh 364 Aug 9 16:35 CHANGES
-rw------- 1 yoh yoh 138 Aug 9 16:35 README
-rw------- 1 yoh yoh 633 Aug 9 16:35 dataset_description.json
-rw------- 1 yoh yoh 405 Aug 9 16:35 participants.json
-rw------- 1 yoh yoh 49 Aug 9 16:35 participants.tsv
-r-------- 1 yoh yoh 2881 Aug 9 16:35 run001.json
-r-------- 1 yoh yoh 111107 Aug 9 16:35 run001.nii.gz
-rw------- 1 yoh yoh 339 Aug 9 16:35 scans.json
/tmp/out/run001.json: "BidsGuess": [
/tmp/out/run001.json- "derived",
/tmp/out/run001.json- "_acq-epse2_dir-AP_run-20_dwi"
/tmp/out/run001.json-],
so looks all good and legit. I guess we need to look into fixing nibabel to not overzealosly remove there since in this particular case it looks like a legit DWI (dcm_qa_xa30/In/20_DWI_dir80_AP
) and not derived images right @neurolabusc ?
@yarikoptic 20_DWI_dir80_AP is a derived image. This is clear in the JSON:
"ImageType": ["DERIVED", "PRIMARY", "DIFFUSION", "TRACEW"],
"RawImage": false,
The derived TRACE image is typically the geometric mean of the raw directional images. One should expect better derived diffusion measures (TRACE, MD, ADC, FA, etc) after processing with dwidenoise, mrdegibbs, topup and eddy. In general, BIDS treats derivatives differently than source data.
What exactly is the fix for this? I'm experiencing the same thing.
no fix yet unfortunately... I followed up on https://github.com/nipy/nibabel/issues/1245
Hi I'm getting the same issue. Did anyone resolving this. Running through Singularity container.
Error: numpy.exceptions.AxisError: axis 1 is out of bounds for array of dimension 1
a solution is proposed by @effigies in https://github.com/nipy/nibabel/issues/1245#issuecomment-1960388301 . Those who can -- try by using patched nibabel pip install git+https://github.com/effigies/nibabel.git@fix/dicom-missing-frame-indices
if you do
pip install git+https://github.com/effigies/nibabel.git@fix/dicom-missing-frame-indices
pip install pydicom
then you could test on your sample dicom using smth like python -W ignore::UserWarning -c 'import sys; f=sys.argv[1]; print(f, end=" "); from nibabel.nicom import dicomwrappers as didw; print(didw.wrapper_from_file(f).image_shape)' FILENAME
@yarikoptic Thanks. I'll try that and let you know.
The https://github.com/nipy/nibabel/issues/1245#issuecomment-1960388301 fix has resolved this (locally).
Would you be able to update this patch into a new heudiconv container? I'm not sure how to fix the container myself.
nibabel 5.2.1 is on PyPI.
I have now merged and released
so you should be all set to use that docker image
❯ docker pull nipy/heudiconv
Using default tag: latest
latest: Pulling from nipy/heudiconv
09e2bc8a597c: Already exists
119c26737198: Already exists
55899c7a837b: Already exists
d976abd2a5fb: Already exists
11b5f8f00533: Already exists
5e3ef4f14ec4: Pull complete
d89fd4dd54dd: Pull complete
7ab0c18a5b97: Pull complete
99e911c23a9d: Pull complete
148a08b8543f: Pull complete
Digest: sha256:5f26567ac56461ddaf734034a92c54eee1203236f512051d8070630b64e174d8
Status: Downloaded newer image for nipy/heudiconv:latest
docker.io/nipy/heudiconv:latest
❯ docker run --rm nipy/heudiconv --version
1.1.0
❯ docker run --rm --entrypoint /opt/miniconda-py39_4.12.0/bin/python nipy/heudiconv:master -c 'import nibabel; print(nibabel.__version__)'
5.2.1
Summary
Description of issue: using heudiconv to convert some participants' sessions ("vis") results in a
numpy.AxisError: axis 1 is out of bounds for array of dimension 1
error. The same heuristic was successfully used on other participants, but I'm not seeing the difference between participant data that is successfully converted and the data that runs into the error.commands:
command-line output:
heuristic:
Platform details:
Choose one:
[ ] Local environment
[x] Container
Docker tag: nipy/heudiconv:latest image ID: 0d55d3d24064
Heudiconv version: 0.10.0