(7FE0,0010) Pixel Data has an undefined length indicating that it's compressed, but the data isn't encapsulated as required

[hari-krishna2000]

ValueError(
                "(7FE0,0010) Pixel Data has an undefined length indicating "
                "that it's compressed, but the data isn't encapsulated as "
                "required. See pydicom.encaps.encapsulate() for more "
                "information"
            )

We have a few dicoms we would like to process, and when we write_file the Dataset for these dicoms ang, we see the above exception being raised.

pydicom | 2.4.0

(0028, 0002) Samples per Pixel                   US: 1
(0028, 0004) Photometric Interpretation          CS: 'MONOCHROME2'
(0028, 0010) Rows                                US: 2829
(0028, 0011) Columns                             US: 2389
(0028, 0034) Pixel Aspect Ratio                  IS: [1, 1]
(0028, 0100) Bits Allocated                      US: 16
(0028, 0101) Bits Stored                         US: 12
(0028, 0102) High Bit                            US: 11
(0028, 0103) Pixel Representation                US: 0
(0028, 0106) Smallest Image Pixel Value          US: 1
(0028, 0107) Largest Image Pixel Value           US: 3602
(0028, 1050) Window Center                       DS: '2048.0'
(0028, 1051) Window Width                        DS: '4096.0'
(0028, 1052) Rescale Intercept                   DS: '0.0'
(0028, 1053) Rescale Slope                       DS: '1.0'
(0028, 1054) Rescale Type                        LO: 'US'
(7fe0, 0010) Pixel Data                          OW: Array of 13516962 elements

[scaramallion]

Your Transfer Syntax UID is 1.2.840.10008.1.2.4.80 - JPEG-LS Lossless Image Compression, which is a compressed transfer syntax, but your Pixel Data is uncompressed. Either compress the Pixel Data to match the transfer syntax or use an uncompressed transfer syntax.

[hari-krishna2000]

If the pixel data is uncompressed, then it should not have is_undefined_length = True. Why is this being set to True for the failing dicoms?

[scaramallion]

The data also doesn't look like its anonymised, please don't post unanonymised data

[scaramallion]

I don't know without seeing your code. Either the original data is encoded incorrectly or you're doing something wrong.

[hari-krishna2000]

dataset = pydicom.dcmread(save_path)
    for data_element in dataset:
        if data_element.name not in copied_tags:
            data_element.value = ''
    dataset.remove_private_tags()
    dataset.save_as(save_path, write_like_original=False)`

The above exception is raised at dataset.save_as

[hari-krishna2000]

a simple dcmread on the dicom file path

[scaramallion]

It looks more likely that the original dataset is encoded incorrectly. After dcmread() try:

from pydicom.uid import ImplicitVRLittleEndian

dataset = pydicom.dcmread(save_path)
dataset.file_meta.TransferSyntaxUID = ImplicitVRLittleEndian
# rest of your code

That should hopefully fix any dodgy encoding in Pixel Data, but only for those dodgy datasets!

[hari-krishna2000]

I am able to run save_as after making the above change but the resultant DICOM is totally blank. It is known that the original DICOM was not blank, because Simple-ITK was able to read the image to provide an image file from the dicom

[scaramallion]

from pydicom import dcmread
from pydicom.pixel_data_handlers.util import get_expected_length
from pydicom.uid import ImplicitVRLittleEndian

ds = dcmread("0bc7cd59-f03861ab-f1720e77-294c2224-24462647")
if (
    ds.file_meta.TransferSyntaxUID.is_compressed  
    and get_expected_length(ds) == len(ds.PixelData)
):
    ds.file_meta.TransferSyntaxUID = ImplicitVRLittleEndian
    ds.is_implicit_VR = True

ds.save_as("out.dcm")

Looks OK to me:

[scaramallion]

So the issue seems solely to be datasets that have been written using an incorrect Transfer Syntax UID

[hari-krishna2000]

What viewer are you using? I am not able to view using Horos or Weasis.

[scaramallion]

ImageJ

[scaramallion]

Although now that I check it, DCMTK is having a fit with the output so maybe there's another issue... let me have another look.

[scaramallion]

OK, try with Explicit VR Little Endian instead:

from pydicom import dcmread
from pydicom.pixel_data_handlers.util import get_expected_length
from pydicom.uid import ExplicitVRLittleEndian

ds = dcmread("0bc7cd59-f03861ab-f1720e77-294c2224-24462647")
if (
    ds.file_meta.TransferSyntaxUID.is_compressed  
    and get_expected_length(ds) == len(ds.PixelData)
):
    ds.file_meta.TransferSyntaxUID = ExplicitVRLittleEndian

ds.save_as("out.dcm")

There's something a little bit odd going on with the Implicit VR conversion (namely that it's not actually implicit VR for some reason). I didn't think we needed to set Dataset.is_implicit_VR when the transfer syntax is set...?

[hari-krishna2000]

Setting TransferSyntaxUID to ExplicitVRLittleEndian worked like a charm, thanks! Also as you mentioned setting TransferSyntaxUID to ImplicitVRLittleEndian, then specifically setting is_implicit_VR to True makes it show up on Horos/Weasis.

[hari-krishna2000]

What could be the root cause for this issue? I would assume that if this scan were produced, it would have the right DataElements?

[scaramallion]

The Transfer Syntax UID in the encoded file is wrong. When you have a compressed transfer syntax such as JPEG the Pixel Data element is supposed to be encoded differently. However the pixel data is encoded as if the transfer syntax were uncompressed - as it should be because the pixel data itself hasn't been compressed.

When you save it, pydicom checks to see if the pixel data is uncompressed and compares that with the set transfer syntax. Since they don't match you get the exception.

[scaramallion]

I've created an issue for the Implicit VR not working as I'd expect, you shouldn't need to set Dataset.is_implicit_VR manually like that.