dcm2niix node does not error out whenever dcm2niix exits with 1

Summary

Originally observed by @dnkennedy while running heudiconv on what appears to be incomplete series. then reproduced on some shorter case of 3 slices dicoms picked up from 10k of them (can't share, didn't find any public ones).

Actual behavior

The point is that if we have

❯ cd /tmp/ttt
MR.1.3.12.2.1107.5.2.43.167017.2020012914293355999131660  MR.1.3.12.2.1107.5.2.43.167017.2020012914301930881739443  MR.1.3.12.2.1107.5.2.43.167017.2020012914320550634554326
❯ dcm2niix -z y .
Chris Rorden's dcm2niiX version v1.0.20220720  (JP2:OpenJPEG) GCC12.1.0 x86-64 (64-bit Linux)
Found 3 DICOM file(s)
Slices stacked despite coil variation 'H1' vs 'H7' (use '-m o' to turn off merging)
Warning: Interslice distance varies in this volume (incompatible with NIfTI format).
Warning: Missing images? Expected 3 images, but instance number (0020,0013) ranges from 23 to 136
Convert 3 DICOM as ./_swi_acq-QSMX3echos_20200129132019_26_cH11_ph (264x288x3x1)
Warning: Unable to rotate 3D volume: slices not equidistant: 40 != 73
Compress: "/usr/bin/pigz" -n -f -6 "./_swi_acq-QSMX3echos_20200129132019_26_cH11_ph.nii"
❯ echo $?
1

reproducing execution with nipype:

import random
from nipype import Node
from nipype.interfaces.dcm2nii import Dcm2niix
import sys
from glob import glob

folder = sys.argv[-1]
files = glob(f"{folder}/*")
print(f"Working on {folder} with {len(files)}")

fromfile = None
convertnode = Node(Dcm2niix(from_file=fromfile), name="convert")
convertnode.base_dir = "/tmp"
convertnode.inputs.source_names = files
convertnode.inputs.out_filename = "/tmp/dcm2niixout%03d" % random.randint(0, 999)
convertnode.terminal_output = "allatonce"
convertnode.inputs.bids_format = True
eg = convertnode.run()
print(f"returncode={eg.runtime.returncode}")

which produces

❯ python /tmp/nipype_convert.py /tmp/ttt; echo $?; ls -l /tmp/ttt
Working on /tmp/ttt with 3
230721-10:38:18,884 nipype.workflow INFO:
     [Node] Setting-up "convert" in "/tmp/convert".
230721-10:38:18,886 nipype.workflow INFO:
     [Node] Outdated cache found for "convert".
230721-10:38:18,888 nipype.workflow INFO:
     [Node] Executing "convert" <nipype.interfaces.dcm2nii.Dcm2niix>
230721-10:38:18,918 nipype.interface INFO:
     stdout 2023-07-21T10:38:18.918151:Chris Rorden's dcm2niiX version v1.0.20220720  (JP2:OpenJPEG) GCC12.1.0 x86-64 (64-bit Linux)
230721-10:38:18,918 nipype.interface INFO:
     stdout 2023-07-21T10:38:18.918151:Found 3 DICOM file(s)
230721-10:38:18,918 nipype.interface INFO:
     stdout 2023-07-21T10:38:18.918151:Slices stacked despite coil variation 'H11' vs 'H7' (use '-m o' to turn off merging)
230721-10:38:18,918 nipype.interface INFO:
     stdout 2023-07-21T10:38:18.918151:Warning: Interslice distance varies in this volume (incompatible with NIfTI format).
230721-10:38:18,918 nipype.interface INFO:
     stdout 2023-07-21T10:38:18.918151:Warning: Missing images? Expected 3 images, but instance number (0020,0013) ranges from 23 to 136
230721-10:38:18,918 nipype.interface INFO:
     stdout 2023-07-21T10:38:18.918151:Convert 3 DICOM as ./tmp/dcm2niixout658_cH11_ph (264x288x3x1)
230721-10:38:18,924 nipype.interface INFO:
     stdout 2023-07-21T10:38:18.924369:Warning: Unable to rotate 3D volume: slices not equidistant: 40 != 73
230721-10:38:18,924 nipype.interface INFO:
     stdout 2023-07-21T10:38:18.924369:Compress: "/usr/bin/pigz" -n -f -6 "./tmp/dcm2niixout658_cH11_ph.nii"
230721-10:38:18,930 nipype.workflow INFO:
     [Node] Finished "convert", elapsed time 0.024807s.
returncode=1
0
total 828
-r-------- 1 yoh yoh 282446 Jul 21 10:17 MR.1.3.12.2.1107.5.2.43.167017.2020012914293355999131660
-r-------- 1 yoh yoh 282444 Jul 21 10:17 MR.1.3.12.2.1107.5.2.43.167017.2020012914301930881739443
-r-------- 1 yoh yoh 282440 Jul 21 10:17 MR.1.3.12.2.1107.5.2.43.167017.2020012914320550634554326

so -- it runs, dcm2niix exits with 1, nipype does not fail it, as it does e.g. when dcm2niix exits with 2:

❯ python /tmp/nipype_convert.py /tmp/bogus
Working on /tmp/bogus with 1
230721-10:40:31,406 nipype.workflow INFO:
     [Node] Setting-up "convert" in "/tmp/convert".
230721-10:40:31,407 nipype.workflow INFO:
     [Node] Outdated cache found for "convert".
230721-10:40:31,409 nipype.workflow INFO:
     [Node] Executing "convert" <nipype.interfaces.dcm2nii.Dcm2niix>
230721-10:40:31,439 nipype.interface INFO:
     stdout 2023-07-21T10:40:31.439841:Chris Rorden's dcm2niiX version v1.0.20220720  (JP2:OpenJPEG) GCC12.1.0 x86-64 (64-bit Linux)
230721-10:40:31,439 nipype.interface INFO:
     stderr 2023-07-21T10:40:31.439955:Error: Unable to find any DICOM images in /tmp/convert (or subfolders 5 deep)
230721-10:40:31,446 nipype.workflow INFO:
     [Node] Finished "convert", elapsed time 0.015616s.
230721-10:40:31,446 nipype.workflow WARNING:
     [Node] Error on "convert" (/tmp/convert)
Traceback (most recent call last):
  File "/tmp/nipype_convert.py", line 18, in <module>
    eg = convertnode.run()
         ^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/heudiconv/heudiconv-master/venvs/dev3/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 527, in run
    result = self._run_interface(execute=True)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/heudiconv/heudiconv-master/venvs/dev3/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 645, in _run_interface
    return self._run_command(execute)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/heudiconv/heudiconv-master/venvs/dev3/lib/python3.11/site-packages/nipype/pipeline/engine/nodes.py", line 771, in _run_command
    raise NodeExecutionError(msg)
nipype.pipeline.engine.nodes.NodeExecutionError: Exception raised while executing Node convert.

Cmdline:
    dcm2niix -b y -z y -x n -t n -m n -f /tmp/dcm2niixout592 -o . -s n -v n /tmp/convert
Stdout:
    Chris Rorden's dcm2niiX version v1.0.20220720  (JP2:OpenJPEG) GCC12.1.0 x86-64 (64-bit Linux)
Stderr:
    Error: Unable to find any DICOM images in /tmp/convert (or subfolders 5 deep)
Traceback:
    RuntimeError: subprocess exited with code 2.

Expected behavior

have an exception similar to when exits with code 2

How to replicate the behavior

see above by BYOBD (bring your own broken dicom)

https://github.com/nipy/nipype/blob/ca27e5187e3c12be5b5565c73be40f7ae70c1689/nipype/interfaces/dcm2nii.py#L449-L450

IIRC, dcm2niix uses (used?) 1 return codes for warnings (but conversion still occurred), and 2+ for serious problems where conversion was not possible.

might be... from https://github.com/rordenlab/dcm2niix/issues/733#issuecomment-1646261249 by @neurolabusc

Exit code 0 is for success, exit code 1 is the undefined error. Other exit codes are for clearly defined errors.

so I guess I will quickly send a PR.

https://github.com/nipy/nipype/blob/ca27e5187e3c12be5b5565c73be40f7ae70c1689/nipype/interfaces/dcm2nii.py#L449-L450

IIRC, dcm2niix uses (used?) 1 return codes for warnings (but conversion still occurred), and 2+ for serious problems where conversion was not possible.

seems to be not the case: when we took a philips dicoms series and removed only 10 (out of 60) of the last dynamic via

/bin/ls -1 *dcm | sort -n --key 3 -t- | tail -n 10 | xargs rm

dcm2niix v1.0.20220720 exited with 1 and no .nii.gz was produced:

❯ dcm2niix -z y -b y *dcm && echo allgood || echo "error $?"
Chris Rorden's dcm2niiX version v1.0.20220720  (JP2:OpenJPEG) GCC13.2.0 x86-64 (64-bit Linux)
Warning: only processing last of 22970 input files (recompile with 'myEnableMultipleInputs' to recursively process multiple files)
Found 22970 DICOM file(s)
DICOM images may be missing, expected 60 spatial locations per volume, but found 22970 slices.
Slice positions repeated, but number of slices (22970) not divisible by number of repeats (383): missing images?
Hint: expected 60 locations
Error: Check sorted order: 4D dataset has 1 volumes, but volume index ranges from 1..383
Warning: Interslice distance varies in this volume (incompatible with NIfTI format).
Unable to equalize slice distances: slice order not consistently ascending.
First spatial position repeated 383 times
Error:  Recompiling with '-DmyInstanceNumberOrderIsNotSpatial' might help.
dcm2niix -z y -b y *dcm  4.66s user 0.39s system 99% cpu 5.059 total
error 1

❯ lst
total 4
lrwxrwxrwx 1 yoh yoh 124 Dec  7 13:30 1.3.46.670589.11.23033.5.0.6668.2017120710301462010-701-22921-1ls1fpe.dcm -> .git/annex/objects/4v/qk/MD5E-s27668--8c4ab9ec8852113d3b502df2cfac43d0.dcm/MD5E-s27668--8c4ab9ec8852113d3b502df2cfac43d0.dcm*
...

processing the entire sequence with all volumes exits with 0 and has following output

```shell ❯ dcm2niix -z y -b y *dcm && echo allgood || echo "error $?" Chris Rorden's dcm2niiX version v1.0.20220720 (JP2:OpenJPEG) GCC13.2.0 x86-64 (64-bit Linux) Warning: only processing last of 22980 input files (recompile with 'myEnableMultipleInputs' to recursively process multiple files) Found 22980 DICOM file(s) Philips Scaling Values RS:RI:SS = 0.869353:0:0.0562237 (see PMC3998685) Convert 22980 DICOM as /tmp/ABCD/ABCD-Like-Subject/rest/DICOM/DICOM_rsfMRI_1_20171207103014_701 (96x96x60x383) Compress: "/usr/bin/pigz" -b 960 -n -f -6 "/tmp/ABCD/ABCD-Like-Subject/rest/DICOM/DICOM_rsfMRI_1_20171207103014_701.nii" Conversion required 6.737152 seconds (4.909542 for core code). dcm2niix -z y -b y *dcm 32.28s user 1.03s system 485% cpu 6.863 total allgood ```

and removing entire last dynamic (60 files) causes a Warning but then exits with 0

```shell ❯ dcm2niix -z y -b y *dcm && echo allgood || echo "error $?" Chris Rorden's dcm2niiX version v1.0.20220720 (JP2:OpenJPEG) GCC13.2.0 x86-64 (64-bit Linux) Warning: only processing last of 22920 input files (recompile with 'myEnableMultipleInputs' to recursively process multiple files) Found 22920 DICOM file(s) Warning: Expected 382 volumes but found spatial position repeats 383 times. Error: Check sorted order: 4D dataset has 382 volumes, but volume index ranges from 1..383 Philips Scaling Values RS:RI:SS = 0.869353:0:0.0562237 (see PMC3998685) Convert 22920 DICOM as /tmp/ABCD/ABCD-Like-Subject/rest/DICOM/DICOM_rsfMRI_1_20171207103014_701 (96x96x60x382) Compress: "/usr/bin/pigz" -b 960 -n -f -6 "/tmp/ABCD/ABCD-Like-Subject/rest/DICOM/DICOM_rsfMRI_1_20171207103014_701.nii" Conversion required 6.526654 seconds (4.682719 for core code). dcm2niix -z y -b y *dcm 33.81s user 0.92s system 523% cpu 6.632 total allgood ```

To clarify, dcm2niix returns a 0 on success, and non-zero for failure. The error codes are here:

EXIT_FAILURE (unspecified)
kEXIT_NO_VALID_FILES_FOUND
kEXIT_REPORT_VERSION
kEXIT_CORRUPT_FILE_FOUND
kEXIT_INPUT_FOLDER_INVALID
kEXIT_OUTPUT_FOLDER_INVALID
kEXIT_OUTPUT_FOLDER_READ_ONLY
kEXIT_SOME_OK_SOME_BAD
kEXIT_RENAME_ERROR
kEXIT_INCOMPLETE_VOLUMES_FOUND //issue 515
kEXIT_NOMINAL //did not expect to convert files

You can see that removing a slice does cause an error (exit 1). In a situation like this, dcm2niix will return exit 10 if the DICOM files explicitly report the number of volumes in the series, but 1 if it is unclear if this was intended.

git clone git@github.com:neurolabusc/dcm_qa_philips.git
cd dcm_qa_philips/In/Bangalore_2008/dwi_no-phi
rm IM_2040
dcm2niix -f %s ./
Chris Rorden's dcm2niiX version v1.0.20231111  Clang14.0.3 ARM (64-bit MacOS)
Found 2039 DICOM file(s)
DICOM images may be missing, expected 60 spatial locations per volume, but found 2039 slices.
Slice positions repeated, but number of slices (2039) not divisible by number of repeats (34): missing images?
...
$ echo $?
$ 1

Removing an entire volume will typically not generate an error, as most DICOM files do not explicitly report the number of volumes expected in the series. One would expect identical DICOMs for a series with 59 volumes as from a series with 60 volumes where the sequence was aborted before the last volume was acquired.

the problem here is that we have behavior of dcm2niix depending on when/how operator interrupted the scan -- if the full dynamic was acquired, (but not all dynamics), dcm2niix issues just an Error to stderr but completes the conversion and exits with 0. So, once in 60 (numbers of slices) cases we would succeed when otherwise would fail.

@yarikoptic dcm2niix is limited to information from the files it is provided. I suggest you work with your Philips clinical scientist to see if the number of volumes can be included in future DICOMs. Happy to update dcm2niix if this information is encoded.

nipy / nipype