Failed conversion with Error: Check sorted order: 4D dataset has 1 volumes, but volume index ranges from 1..###

[ericearl]

Describe the bug

Philips DICOMs from ABCD Study 2 year follow up visit fail conversion with:

Error: Check sorted order: 4D dataset has 1 volumes, but volume index ranges from 1..###

Where ### is either 383 or 445.

To reproduce

Steps to reproduce the behavior:

1. Run the command dcm2niix -b y -ba y -z y -f %3s_%f_%p_%t -o /data/BIDS/tmp_dcm2bids/sub-NDARINVANONYMIZED_ses-2YearFollowUpYArm1 /data/DICOM/sub-NDARINVANONYMIZED/ses-2YearFollowUpYArm1
2. 4 Errored series show up with the above messages, but I can't tell from the logs which series order they were attempting to convert.
3. Notice the absence of some series in the tmp_dcm2bids subfolder.
4. Most likely series in this case are the first run of the SST fMRI task and runs 1, 3, and 4 of the resting-state fMRI.

Expected behavior

I was expecting both more log messages around what was attempting to convert and which DICOM(s) misled the command or caused the conversions to fail.

Output log

Most of the log contains subfolders and filenames named with exact acquisition times as well as exact NDA GUIDs. I cannot share most of the logs.

Version

I tried with both:

1. pip-installed v1.0.20220715: dcm2niiX version v1.0.20220505 GCC7.5.0 x86-64 (64-bit Linux)
2. GitHub release downloaded v1.0.20240202: v1.0.20240202 (JP2:OpenJPEG) (JP-LS:CharLS) GCC8.4.0 x86-64 (64-bit Linux)

Troubleshooting

• The latest stable release does not resolve my issue on Linux 64-bit.
• I have not yet tried the latest commit on the development branch
• I can't directly give you the DICOM data, but I can guide you to download it.

[ericearl]

Another small discovery: when I remove the first "volume" by deleting the 445 or 383 DICOMs that appear sequentially first in each stack of 60 (I think I said that right?) and then run dcm2niix, it has no problems.

I think this is due to a "corrupt volume" as described by https://github.com/DCAN-Labs/abcd-dicom2bids/blob/main/src/remove_RawDataStorage_dcms.py as a series whose first DICOM has =RawDataStorage in field (0002,0002). Whatever is corrupt about it is what I really don't understand.

[neurolabusc]

@ericearl I am unable to replicate this issue. Feel free to submit a pull request if you track down the issue. Alternatively, share the data with me institutional email. You can always replace tags from DICOM data to protect privacy:

gdcmanon --dumb 
  --replace 0010,0010,Annon \
  --replace 0010,0020,02 \
  --replace 0008,1030,Annon \
  --replace 0010,0040,F \
  --replace 0010,1010,42 \
  --replace 0010,1030,75 \
  --replace 0010,0030,19660101 \
  --replace 0020,0010,433724515 \
  --replace 0008,0022,20130717 \
  --replace 0008,0023,20130717 \
  --replace 0008,0033,142035.93 \
  -i ... -o ...

[ericearl]

And another small discovery: I can get all of the volumes to convert independently using dcm2niix. Most of them are 60 slices in the Z direction, but one is 59 slices in the Z direction (I'm guessing this is the one volume with the one slice having "RawDataStorage" in DICOM field 0002,0002).

Is this why when dcm2niix is trying to convert them altogether as one 4D volume it fails? I guess the options are to either exclude the 60th slice from every volume or to exclude the first volume containing the bad slice from the 4D timeseries. What do you think is the appropriate action to take?

For what it's worth, I isolated the series and ran dcmniix and got this:

> dcm2niix -b y -ba y -z y -f %3s_%f_%p_%t -o ../ABCD_Philips_2YearFollowUpYArm1_rsfMRI_nifti .
Chris Rorden's dcm2niiX version v1.0.20220505  GCC7.5.0 x86-64 (64-bit Linux)
Found 22980 DICOM file(s)
Skipping non-image DICOM: ./sub-NDARINVANONYMIZED_ses-2YearFollowUpYArm1_dicom000001.dcm
DICOM images may be missing, expected 60 spatial locations per volume, but found 22979 slices.
Slice positions repeated, but number of slices (22979) not divisible by number of repeats (382): missing images?
Hint: expected 60 locations
Error: Check sorted order: 4D dataset has 1 volumes, but volume index ranges from 1..383
Warning: Interslice distance varies in this volume (incompatible with NIfTI format).
Unable to equalize slice distances: slice order not consistently ascending.
First spatial position repeated 383 times
Error:  Recompiling with '-DmyInstanceNumberOrderIsNotSpatial' might help.

[neurolabusc]

This sounds like dcm2niix is exhibiting the desired behavior. Classic DICOM stores each 2D slice as a separate file. The NIfTI standard requires that all 3D volumes in a 4D time series have the same number of slices. If a scan was aborted part way through a volume, or 2D slices were lost during data transfer, some human intervention is required. If slices at the start of the series are lost, event times for fMRI tasks must be adjusted accordingly.

For most manufacturers, you can use dcm2niix's rename function (-r y) to give your DICOM human readable file names, for example having the filename include the instance number (`-f %r.dcm). You can then discard the leading or trailing slices required to make consistent volumes:

dcm2niix -r y -f %t/%s_%p/%4r_%o.dcm /path/to/partial4D/DICOMs

The one problem with this approach is that your data comes from a Philips scanner. While other manufacturers use sequential instance numbers, in my experience Philips 2D slices are often assigned random instance numbers (perhaps reflecting the order they returned from a parallel reconstruction system). This is legal (DICOM does not require sequential or even unique instance numbers).

Philips usually stores non-image sequence information (exam cards) as DICOM RawDataStorage files using some proprietary encoding. This is also legal: DICOM can be used for non-image data. However, in practice this causes issues, these files are often not saved, Philips has not documented how to read them, and they confuse many users. I suspect if you check the provenance of your images, you will find that one of the image slices was erroneously overwritten by this non-image data at some stage in the chain of archiving. I suggest working with the ABCD site and the Philips Clinical Scientist affiliated with the site to ensure that the retained DICOMs are of archival quality.

I would also suggest Philips users and those considering purchasing Philips MRI systems to lobby the manufacturer to follow a few conventions. While their files are legal, they do have features that are not competitive. It is helpful if series numbers are sequential. For reproducible science details like slice timing, readout time and phase encoding polarity should be stored in the DICOM header with the image, ideally using the public tags. For enhanced DICOM, header redundancy should be eliminated. Comparing the lean enhanced DICOMs from Siemens and Canon to the huge files created by Philips drives this home: Philips images slow down transfers, overwhelm PACS discs and attempts to inspect the headers will crash many popular tools.

[ericearl]

@neurolabusc Thank you for the wealth of helpful information! I will advocate for proper use of the Philips DICOMs, but I am not a part of the ABCD consortium anymore since I moved labs in April 2021. Perhaps we should get Don Hagler or someone else at the ABCD Study's DAIRC to weigh in here and see their thoughts? I can think of a few folks that might have good thoughts on this.

Was the above information from me enough to mean it would not be helpful to anonymize the DICOMs and send them your way? I think I've figured out how to use DicomCleaner well enough to anonymize them safely now. Or maybe I could send just an anonymous version of the offending slice?

If we're good here, I could close this issue, but if there's more to do I'm happy to leave it open and get some more opinions from ABCD DAIRC folks.

By the way, you were right about trying to open the offending DICOM RawDataStorage "slice". The DICOM viewer I was trying (Mango on Windows) couldn't load it.