dcm2niix fails

[opennog]

Describe the bug When I run:

$ dcm2niix dcm2niix_fails The output is the following: Chris Rorden's dcm2niiX version v1.0.20211006 (JP2:OpenJPEG) (JP-LS:CharLS) GCC7.5.0 x86-64 (64-bit Linux) Found 208 DICOM file(s) Warning: Assuming 7FE0,0010 refers to an icon not the main image Warning: Assuming 7FE0,0010 refers to an icon not the main image Warning: Assuming 7FE0,0010 refers to an icon not the main image ... Warning: Assuming 7FE0,0010 refers to an icon not the main image No valid DICOM images were found

Thus dcm2niix fails to convert the provided image to nifti.

The problem comes because the tag (0088,0200) is included in the header and if you remove it from every single dicom file and run the dcm2niix again it will work. However, I hypothesize that having that tag present is not the source of the data. Since I've seen other MRI volumes in which that tag is present and dcm2niix does work.

I receive this data through a PACS application based on python. In particular it uses the lib PynetDicom (https://pydicom.github.io/pynetdicom/stable/#). In this context, I've noted the following:

1) if I use PACS with pynetdicom==1.4.1 - the transferred files will have the following header: Dataset.file_meta ------------------------------- (0002, 0000) File Meta Information Group Length UL: 190 (0002, 0001) File Meta Information Version OB: b'\x00\x01' (0002, 0002) Media Storage SOP Class UID UI: MR Image Storage (0002, 0003) Media Storage SOP Instance UID UI: 2.16.124.113543.6006.99.08911600347172884136 (0002, 0010) Transfer Syntax UID UI: Explicit VR Little Endian (0002, 0012) Implementation Class UID UI: 1.2.826.0.1.3680043.9.3811.1.4.1 (0002, 0013) Implementation Version Name SH: 'PYNETDICOM_141'

and dcm2niix will work.

2) If I use PACS with pynetdicom>=2.0.0 - the transferred files will have the following header: Dataset.file_meta ------------------------------- (0002, 0000) File Meta Information Group Length UL: 188 (0002, 0001) File Meta Information Version OB: b'\x00\x01' (0002, 0002) Media Storage SOP Class UID UI: MR Image Storage (0002, 0003) Media Storage SOP Instance UID UI: 2.16.124.113543.6006.99.07451693937431759783 (0002, 0010) Transfer Syntax UID UI: Implicit VR Little Endian (0002, 0012) Implementation Class UID UI: 1.2.826.0.1.3680043.9.3811.2.0.2 (0002, 0013) Implementation Version Name SH: 'PYNETDICOM_202'

and dcm2niix will not work.

So I hypothesize the problem might come from the Transfer Syntax of these latter image. Does dcm2niix has full support of Implicit VR Little Endian? dcm2niix_fails.zip dcm2niix_works.zip

I attach 2 zip files, they can be used to reproduce the issue. Ofc, dcm2niix_fails.zip should fail and dcm2niix_works.zip should work :)

Finally, as you indicate, I've made the same experiment with the latest commit and dcm2niix still fails. See summarized output bellow:

$ ./dcm2niix ~/Downloads/dcm2niix_fails Chris Rorden's dcm2niiX version v1.0.20220915 GCC9.4.0 x86-64 (64-bit Linux) Found 208 DICOM file(s) Warning: Assuming 7FE0,0010 refers to an icon not the main image Warning: Assuming 7FE0,0010 refers to an icon not the main image Warning: Assuming 7FE0,0010 refers to an icon not the main image .... Warning: Assuming 7FE0,0010 refers to an icon not the main image No valid DICOM images were found

Thank you very much, Oscar

[neurolabusc]

This is an issue with the ADNI images, and not dcm2niix. You should contact the ADNI team to correct this problem at the source.

The images were touched by a GE GEIIS PACS system as described in issue 367 which inserted a thumbnail image. While thumbnails are allowed in the standard, the GE implementation violated the standard (e.g. thumbnails MUST use the same transfer syntax and bits allocated). The ADNI team ran some DICOM anonymization and cleanup routines that further obfuscate the issue. The DICOM standard is very complex, and dcm2niix and many other tools assume that the images are valid.

[opennog]

Thank you very much for your quick response.

I looked for ADNI data in order to be able to share it, but I've originally experienced it with Siemens, GE, Philips MRI data from different US hospitals that unfortunately I cannot share.

This ADNI data in particularly is from Siemens, and this images has not been touched by GE GEIIS PACS since both of the volumes I'm sharing are the same, although they have been transferred using different versions of pynetdicom.

Finally, this is the ICON Image sequence Tag of both of the above files:

(0088, 0200) Icon Image Sequence 1 item(s) ---- (0028, 0002) Samples per Pixel US: 1 (0028, 0004) Photometric Interpretation CS: 'PALETTE COLOR' (0028, 0010) Rows US: 64 (0028, 0011) Columns US: 64 (0028, 0100) Bits Allocated US: 8 (0028, 0101) Bits Stored US: 8 (0028, 0102) High Bit US: 7 (0028, 0103) Pixel Representation US: 0 (0028, 1101) Red Palette Color Lookup Table Desc US: [256, 0, 8] (0028, 1102) Green Palette Color Lookup Table De US: [256, 0, 8] (0028, 1103) Blue Palette Color Lookup Table Des US: [256, 0, 8]

Apparently, it does not seem to have any transferred syntax. How do you know it is not in agreement with that of the main image? then, what's the explanation why, while both volumes have the same SQ, only one fails dcm2niix conversion?

Thanks

[neurolabusc]

All of these icons are invalid DICOM files, e.g. bits stored for the thumbnail does not match the main image. Most DICOM files are transferred from the scanner to a series of PACS systems. If a buggy GE GEIIS PACS is anywhere in the chain, it will corrupt your DICOM files. This is often hidden by the fact that a subsequent PACS touches the data. You will want to check the provenance of each corrupted image and inform the clinical team to get their GE GEIIS PACS software updated. GE has fixed this problem and they will honor their DICOM conformance statement by upgrading the tool.

dcm2niix includes several kludges to detect these corrupted images, which means that it will succeed with some but not all corrupted images. However, none of these images are valid DICOMs, and none are of archival quality. These kludges impact the maintainability and clarity of the dcm2niix code. This issue has plagued DICOM images for years, and it is unfortunate that GE took over a decade to address it.

[opennog]

Thank you very much for your thorough clarification. I'm still very hesitant this data has gone through GE GEIIS PACS but I understand your point. Do you know if the issue has happened in any other PACS system?

[neurolabusc]

To be clear, the ADNI data fails for a very specific reason: a GE GEIIS PACS inserted an IconImageSequence SQ (0088,0200). A subsequent step removed the actual icon image data (7fe0,0010) but did not remove the SQ.

I suspect your non-ADNI datasets are failing for another reason. In my experience, the DICOM violation created by GE GEIIS PACS is handled differently depending on the subsequent PACS that touches the data. Due to this variability, I have never developed a perfect algorithm to handle the corrupted files. If you can share a sample non-ADNI dataset with my institutional email I might be able to provide more feedback.

I believe the icon issue is specific to the GE GEIIS PACS. However, people should be aware of the way that dcm4che based PACS including professional systems from AGFA mangle any DICOM image they touch.

[neurolabusc]

@opennog I have adapted the code so all the GEIIS icon kludges are ignored if the IconImageSequence SQ (0088,0200) provides an explicit length. This will assume all the icon-specific information is embedded in the SQ. This approach does work with the modified ADNI data, and might work with other datasets that have more well behaved icons but were triggering the kludges designed to handle GEIIS. One idea might be that you have some PACS systems that are generating false positives for my GEIIS patches.

To try this out, get version v1.0.20220923 from the development branch:

$ git clone --branch development https://github.com/rordenlab/dcm2niix
$ cd dcm2niix/console
$ make
$ ./dcm2niix
Chris Rorden's dcm2niiX version v1.0.20220923  Clang12.0.5 ARM (64-bit MacOS)

[neurolabusc]

@captainnova might be nice for the ADNI team to try out this development version. My hope is that this new version will handle more of the ADNI datasets where the GEIIS icon pixel data was removed but where an Icon SQ with explicit length was provided.

[opennog]

Hi @neurolabusc Thanks for reopening the issue. Unfortunately I'm not able to share private data. However, with your later changes I can confirm it works!!!! (see log with both versions bellow)

your commit $ ./dcm2niix ../../MPRAGE Chris Rorden's dcm2niiX version v1.0.20220923 GCC9.4.0 x86-64 (64-bit Linux) Found 176 DICOM file(s) Convert 176 DICOM as ../../MPRAGE/MPRAGE_MPRAGE_20220920104301_2 (256x256x176x1) Compress: "/usr/bin/pigz" -b 960 -n -f -6 "../../MPRAGE/MPRAGE_MPRAGE_20220920104301_2.nii" Conversion required 0.814205 seconds (0.296630 for core code).

old... dcm2niix ../../MPRAGE Chris Rorden's dcm2niiX version v1.0.20211006 (JP2:OpenJPEG) (JP-LS:CharLS) GCC7.5.0 x86-64 (64-bit Linux) Found 176 DICOM file(s) Warning: Assuming 7FE0,0010 refers to an icon not the main image Warning: Assuming 7FE0,0010 refers to an icon not the main image Warning: Assuming 7FE0,0010 refers to an icon not the main image Warning: Assuming 7FE0,0010 refers to an icon not the main image Warning: Assuming 7FE0,0010 refers to an icon not the main image Warning: Assuming 7FE0,0010 refers to an icon not the main image Warning: Assuming 7FE0,0010 refers to an icon not the main image

Thank you very much, I hope in the end you are able to release these changes. If you happen to modify it a bit more don't hesitate to ask me for further testing :)

Best

[neurolabusc]

No need to share the data if the change works. Assuming no one discovers unintended consequences the changes will be part of the next stable release.

[captainnova]

@captainnova might be nice for the ADNI team to try out this development version. My hope is that this new version will handle more of the ADNI datasets where the GEIIS icon pixel data was removed but where an Icon SQ with explicit length was provided.

Hi @neurolabusc, our lab's usual dcm2niix test suite is running now. I've certainly noticed the dangling IconImageSequences from time to time, but did not think to keep a record of their locations for this occasion. I'll search the lab wiki and try to dig some up.

[captainnova]

dcm2niiX version v1.0.20220923 (development branch) passed our usual test suite, and worked with some (non-ADNI) DICOM with the imageless IconImageSequence that had previously caused problems for us. However, it's worth noting that those problems were from manipulating the DICOM with pydicom, not from dcm2niix. dcm2niix v1.0.20211006 also worked just fine with the same DICOM, so the kludges are good, and I'm not experiencing any change except the usual mild BIDS updates.

GE, dcm4che and Agfa aside, it seems to me that the proximal cause of this particular issue is sending the somewhat-mangled DICOM through the PynetDicom PACS and getting more thoroughly mangled DICOM out. Trying to read DICOM that had been rewritten by pydicom is exactly the the problem that triggered the bug report where I found our example bad DICOM. When this sort of flaw goes in pydicom and gets written back out it gets subtly magnified. In our case we decided to either avoid rewrites or completely nuke the IconImageSequence. However, we weren't using it as a PACS. @opennog might be more motivated to do the right thing and report this to the pydicom people.

[neurolabusc]

@captainnova thanks for your rapid check. Based on my design and your experience, I think this patch is now able to handle some DICOMs with icons that previously caused errors without introducing unintended consequences.

[captainnova]

I have to confess that our local pydicom installation has fallen out of date, and is effectively pinned by other things that will take more work to update, so I did not report our issue to pydicom because it would have opened a can of worms we didn't need. I should have suggested @opennog try the latest pynetdicom, and then open an issue if needed.

[opennog]

Thanks @neurolabusc and @captainnova for your explanations. I indeed use the latest versions of both pydicom and pynetdicom. So I'll report them the bug. I'll let you know their findings.