[Bug] Regression handling RGB in XC modality

[fedorov]

Describe the Bug

This sample is no longer rendered correctly: https://viewer.imaging.datacommons.cancer.gov/v3/viewer/?StudyInstanceUIDs=1.3.6.1.4.1.5962.1.2.0.1677425356.1733

For comparison, see how it used to work here: https://github.com/OHIF/Viewers/issues/3354#issuecomment-1673389784:

To download the files in that study (note - it is ~233GB!), assuming you have python and pip on your system, do the following:

pip install --upgrade idc-index
idc download 1.3.6.1.4.1.5962.1.2.0.1677425356.1733

Steps to Reproduce

Load the sample study referenced in the above into the viewer.

The current behavior

First series shows grayscale with multiple slices arranged in a mosaic. Second does not load at all.

The expected behavior

Both should show up in color as used to work in the past given the screenshot in https://github.com/OHIF/Viewers/issues/3354#issuecomment-1673389784

OS

macOS

Node version

n/a

Browser

Chrome

[pedrokohler]

when you say "works", can you confirm if the content returned is compressed or not?

@fedorov when I say it works I mean the proxy doesn't return a 500 error, and given its limit is 32MB it means it gets compressed. The responses come with ~27MB slices as opposed to the 32+MB of when we were going around the proxy.

[fedorov]

Perfect, great news! This also seems to indicate, if the comment in that older issue is accurate, that there is no change in the behavior of the GHC backend.

Next, can you please test this with v1beta1 GHC API?

After that, can you please proceed with testing against the IDC proxy?

@wlongabaugh please remind us which tiers use which version of GHC API. I do not know where to find that information.

[pedrokohler]

Next, can you please test this with v1beta1 GHC endpoint?

Can you provide a server url?

[pedrokohler]

After that, can you please proceed with testing against the IDC proxy?

Hmm... not sure what you mean here. Can you elaborate? I mean, I was testing against a proxy all along. You mean another proxy?

[fedorov]

Sorry I missed that you tested against the proxy. We need to know the behavior with both direct access to GHC and via the proxy. While using proxy, you cannot choose API version, you can only do it while accessing directly, substituting v1 with v1beta1.

Are you able to download that study and make a Google DICOM store for direct testing, or you need help with that?

[pedrokohler]

Are you able to download that study and make a Google DICOM store for direct testing, or you need help with that?

@fedorov I've never created a google DICOM store and I don't have access to IDC's GCP account, which means I'd need help indeed.

note that while accessing IDC DICOM store directly - bypassing the proxy - I am able to visualize both series.

I tried using the IDC fork's config file and the GCP server you sent in the message from the quote above, but I'm getting this error when I try to login

(also I'm not sure that by logging in with the default oidc config I'd be able to access this specific server anyway)

[wlongabaugh]

I was wondering how @pedrokohler would be able to access GHC directly, as he does indeed not have permissions.

The version of the API we are using is in the proxy runtime configuration file that lives in each tier. E.g. for test it in in gs://webapp-deployment-files-idc-test/proxy/proxy_runtime_config.txt.

The dev and test proxies are running v1beta1. Production proxy is still running v1.

[wlongabaugh]

@fedorov OK, I have reread my comment you highlighted above. https://github.com/OHIF/Viewers/issues/4382#issuecomment-2465278258 Yes, I originally thought it was the other way around, that we needed the transfer-syntax key/value to make it work. But yeah, we needed to drop the key/value to make it work. This all makes sense now. That was my first impression of this when it came up, that there was some change in compression that made it work, back at the start of 2024. So drop the transfer-syntax in V3.

[sedghi]

transfer-syntax=* means allow ANY transfer syntax on the server, omitting transfer syntax means LEI

[fedorov]

I do not know what LEI is. But can you give us background why in the past (it appears that) transfer-syntax=* was not used, and later it was added? What triggered that change?

[sedghi]

https://www.dicomlibrary.com/dicom/transfer-syntax/

Explicit VR Little Endian

I believe it's a sensible default for OHIF to follow. It tells the server not to bother with conversion and to send whatever is available, as OHIF can support it. This reduces the server's response time. If there's an issue with using *, it indicates a bug in OHIF.

[wlongabaugh]

The issue is not a bug in OHIF, but the fact that by providing it in the request, Google will not try to compress the 35MB+ slices it sends out.

[fedorov]

This reduces the server's response time. If there's an issue with using *, it indicates a bug in OHIF.

@sedghi The issue in this case is not with OHIF per se, but with the specific combination of OHIF/data/user requirements. To summarize the long thread, we need to have the server compress the data before sending. We have good reasons for that. With the current transfer-syntax=* we cannot satisfy our needs.

Can this be somehow addressed at the OHIF level without us making customizations specific to our fork/use case?

I do not think this use case is limited to IDC.

[fedorov]

@pedrokohler I created a DICOM store that has Visible Female XC study: projects/idc-sandbox-000/locations/us-central1/datasets/fedorov-dev-healthcare/dicomStores/visible-female. I gave you all the necessary permissions. You can use this store to test against DICOM store bypassing proxy, and compare behavior between v1 and v1beta1 GHC API.

[sedghi]

@fedorov if you make sure the server already has compressed data stored for that study then the * should work, no? Can't you call an api in google and ask for change in stored data format?

[fedorov]

First, based on the explanation from @dclunie yesterday, compression in this scenario is done at the HTTP level - it does not change the transfer syntax.

Second, one of our principles in IDC is not to modify the original encoding of the images in the files that are ingested into the DICOM server.

[sedghi]

BTW I added all transfer syntax test data we had here https://github.com/cornerstonejs/cornerstone3D/pull/1568

Seems like we only are not supporting

DeflatedExplicitVRLittleEndianTransferSyntax

[wlongabaugh]

I will reiterate that one way to deal with this is to have the proxy look at the URLs being requested, and if it includes ones of these problematic studies, it can modify the requests headers (ditch transfer-syntax=*) for those studies to kick off the compression.

[fedorov]

I will reiterate that one way to deal with this is to have the proxy look at the URLs being requested

@wlongabaugh in my list of preferred options, increasing complexity of a proxy that is specific to IDC is not at the top. IDC proxy based solution will never benefit other users of OHIF Viewer, and I think there is a good chance that this situation is not unique to IDC. In our case, we need this because of the proxy limitations on the buffer size. In other cases users may want to have smaller buffer size due to limited network bandwidth.

@sedghi would it be possible to add an option for the client/application to choose whether transfer-syntax=* should be included or not?

I hope the answer is yes, and then the client could perhaps estimate the size of PixelData based on frame size and bit depth, and decide based on that whether it should be included or not and request server compression? This could also be controllable via a config option. Just one idea. Maybe there is a better way to know what is the size of the object in the original encoding.

[sedghi]

Yes, setting it is optional, but if you don't, every compressed study will be decompressed, as far as I know. So, perhaps we should have a per-study configuration or something similar.

[dclunie]

@sedghi, you wrote:

Yes, setting it (transfer-syntax=*) is optional, but if you don't, every compressed study will be decompressed, as far as I know.

Short version - I don't think that is a compliant way of receiving compressed frames, even if Google does support it, so you should not be asking for transfer-syntax=*

TL;DNR:

The details here are a digression from the topic at hand (getting back uncompressed XC images, and getting large ones through the proxy by deflate applied at the Content-Encoding rather than Transfer Syntax level), ...

... but it is relevant since it has apparently been observed that requesting multipart/related; type="application/octet-stream"; transfer-syntax=* has an undesirable (and relevant) side effect of blocking Content-Encoding ( e.g., as deflate) as distinct from the simpler (and correct)multipart/related; type="application/octet-stream".

If you request application/octet-stream (whether single part or multipart), as I understand it. the standard says you will get back uncompressed bytes, and the Google documentation implies you will get back uncompressed bytes because the default TS is uncompressed (and it will perform decompression as necessary if it can).

However, I don't think it is legal for a DICOMweb server to return in a frame request a compressed bitstream in an application/octet-stream media type. That media type is only to be used for uncompressed data.

Rather, the server is required to return image/jpeg for JPEG transfer syntaxes, etc. Otherwise how would you know (without inspecting the compressed byte stream) what scheme is used to compress the byte stream in the response, if it was not signaled in the media type in the returned Content-Type header field? See DICOM PS3.18 8.7.3.3.2 Compressed Bulkdata Media Types.

I know that the Google server documentation for retrieval of frames says it supports multipart/related; type="application/octet-stream"; transfer-syntax=* and further that

"For application/octet-stream the bulk data will be returned in whatever format it appears in the uploaded DICOM file"

but I think this is fundamentally wrong; returning compressed byte streams as application/octet-stream is not what the standard specifies; I don't think you should be requesting that, even if it (sometimes) works.

Perhaps you should be requesting multipart/related; type="*/*" or similar, if you don't care how it is compressed or if it is uncompressed, and the Content-Type in the response should tell you what it actually is. Perhaps even multipart/related; type="*/*"; transfer-syntax=*

However there is a warning in the Google documentation about image/jpeg defaulting to a .70 TS that is not supported, so if the images are lossless JPEG such as TS .50, then perhaps type="*/*"; transfer-syntax=* will freak it out.

How is OHIF figuring out what compression scheme is used when requesting multipart/related; type="application/octet-stream"; transfer-syntax=* (perhaps there is a transfer-syntax parameter on the Content-Type of the response that you are cuing off)?

I hope you are not using any TransferSyntaxUID value that might be present in the metadata (if they return Group 0x0002 data elements in the metadata, which they shouldn't) as opposed to (trying to) use QIDO AvailableTransferSyntaxUID in CP 1901. I don't think that Google supports AvailableTransferSyntaxUID though. To state the obvious, if Group 0x0002 TransferSyntaxUID is returned in the metadata, its value will not necessarily match whatever is in the returned pixel data if any transcoding has occurred.

I will do some more experiments wrt. what the Google server does or does not support, but the bottom line is that I think a more robust approach by OHIF to getting back (lossless) compressed pixel data if that is the form it is available in may be required if you have felt the need to use multipart/related; type="application/octet-stream"; transfer-syntax=* for that reason and don't have an alternative pathway already.

I would be very interested to hear if you have experience with DICOMweb servers other than Google's that respond to multipart/related; type="application/octet-stream"; transfer-syntax=* with anything other than uncompressed pixel data.

David

PS. I don't think much has changed in this respect since CP 1509 cleaned up the media types, or even the original description in Sup 161; i.e., I don't think we can blame the PS3.18 reorganization for changing anything in this respect wrt. media types for uncompressed versus compressed data. I will go through the development history of WADO-RS and review the discussions of the media types to see if there is anything I have missed.

[sedghi]

CC @wayfarer3130

[wayfarer3130]

@sedghi, you wrote:

Yes, setting it (transfer-syntax=*) is optional, but if you don't, every compressed study will be decompressed, as far as I know.

Short version - I don't think that is a compliant way of receiving compressed frames, even if Google does support it, so you should not be asking for transfer-syntax=*

TL;DNR:

The details here are a digression from the topic at hand (getting back uncompressed XC images, and getting large ones through the proxy by deflate applied at the Content-Encoding rather than Transfer Syntax level), ...

The deflate CAN be applied at hte content encoding level, but it means to apply default to the multipart/related body as well as to the overall body - that is what static dicomweb does for encoding such images.

... but it is relevant since it has apparently been observed that requesting multipart/related; type="application/octet-stream"; transfer-syntax=* has an undesirable (and relevant) side effect of blocking Content-Encoding ( e.g., as deflate) as distinct from the simpler (and correct)multipart/related; type="application/octet-stream".

If you request application/octet-stream (whether single part or multipart), as I understand it. the standard says you will get back uncompressed bytes, and the Google documentation implies you will get back uncompressed bytes because the default TS is uncompressed (and it will perform decompression as necessary if it can).

However, I don't think it is legal for a DICOMweb server to return in a frame request a compressed bitstream in an application/octet-stream media type. That media type is only to be used for uncompressed data.

Yes, that is actually correct - it should have been multipart/related; type="image/*" with a secondary accept of multipart/related; type="application/octet-stream"; q=0.9

The application/octet-stream is only valid for DICOM encoded responses, and I had conflated that with using it on the frame level retrieve.

Preferring images over application/octet stream, all of them encoded in multipart/related. It would also be acceptable to do the following set: "image/*, application/octet-stream; q=0.9, multipart/related; type="image/*;q=0.8, multipart/related; type="application/octet-stream"; q=0.7 Or, if known that the server supports single part, just the first two.

Rather, the server is required to return image/jpeg for JPEG transfer syntaxes, etc. Otherwise how would you know (without inspecting the compressed byte stream) what scheme is used to compress the byte stream in the response, if it was not signaled in the media type in the returned Content-Type header field? See DICOM PS3.18 8.7.3.3.2 Compressed Bulkdata Media Types.

I know that the Google server documentation for retrieval of frames says it supports multipart/related; type="application/octet-stream"; transfer-syntax=* and further that

"For application/octet-stream the bulk data will be returned in whatever format it appears in the uploaded DICOM file"

but I think this is fundamentally wrong; returning compressed byte streams as application/octet-stream is not what the standard specifies; I don't think you should be requesting that, even if it (sometimes) works.

Perhaps you should be requesting multipart/related; type="*/*" or similar, if you don't care how it is compressed or if it is uncompressed, and the Content-Type in the response should tell you what it actually is. Perhaps even multipart/related; type="*/*"; transfer-syntax=*

Yes, but added several request types with q parameters to indicate preference. I don't know that it will work for image/* but technically it is supposed to work AFAIK.

However there is a warning in the Google documentation about image/jpeg defaulting to a .70 TS that is not supported, so if the images are lossless JPEG such as TS .50, then perhaps type="*/*"; transfer-syntax=* will freak it out.

How is OHIF figuring out what compression scheme is used when requesting multipart/related; type="application/octet-stream"; transfer-syntax=* (perhaps there is a transfer-syntax parameter on the Content-Type of the response that you are cuing off)?

There is the transfer syntax in the response data - which is actually required to distinguish between image/jpeg for .50 and .70

I hope you are not using any TransferSyntaxUID value that might be present in the metadata (if they return Group 0x0002 data elements in the metadata, which they shouldn't) as opposed to (trying to) use QIDO AvailableTransferSyntaxUID in CP 1901. I don't think that Google supports AvailableTransferSyntaxUID though. To state the obvious, if Group 0x0002 TransferSyntaxUID is returned in the metadata, its value will not necessarily match whatever is in the returned pixel data if any transcoding has occurred.

No, that isn't used there. However, the AvailableTransferSyntaxUIDs is used to sub-select the accept header to the comrpessed format if it is provided in the original metadata.

I will do some more experiments wrt. what the Google server does or does not support, but the bottom line is that I think a more robust approach by OHIF to getting back (lossless) compressed pixel data if that is the form it is available in may be required if you have felt the need to use multipart/related; type="application/octet-stream"; transfer-syntax=* for that reason and don't have an alternative pathway already.

I would be very interested to hear if you have experience with DICOMweb servers other than Google's that respond to multipart/related; type="application/octet-stream"; transfer-syntax=* with anything other than uncompressed pixel data.

It is supported fairly generally - I can think of at least 4 PACS systems supporting it.

Bill

PS @dclunie - thanks for responding to that, I had assumed incorrectly that the transfer syntax parameter was supported for this accept header as well as the others, and apparently others have made the same assumption.

[wayfarer3130]

According to 8.7.3.5.1 Multipart Media Types, it is actually legal to request multipart/related; type="application/octet-stream"; transfer-syntax="*" I agree the interpretation of that is a bit iffy, but the request itself is legal according to the syntax.

[sedghi]

Also CCing @amazy the lead for orthanc team, if he can help us with how ohif should send requests

[dclunie]

:) "iffy" is one way to put it - "wrong" is another!

I will draft a CP to add a note to PS3.18 to indicate that returning compressed streams in application/octet-stream is not correct, and that combined with transfer-syntax="*", it should only match uncompressed transfer syntaxes (perhaps including the recent encapsulated uncompressed to get around size limits). Then WG 27 can discus the matter.

[dclunie]

BTW. AvailableTransferSyntaxUIDs should not be in the metadata - it is an instance level search transaction attribute.

[fedorov]

Following this exchange - are there any practical and actionable implications regarding how to proceed with achieving the behavior needed for our use case - requesting compression applied to Content-Encoding?

[amazy]

Also CCing @amazy the lead for orthanc team, if he can help us with how ohif should send requests

Since I'm not a standard expert at all, I'll not comment on the standard itself or on how it should be understood.

I must admit that Orthanc's behavior wrt to multipart/related; type="application/octet-stream"; transfer-syntax="*" was inspired by the Google implementation and it was not incompatible with our understanding of the standard.

That being said, here is how Orthanc works:

• in the /metadata route, Orthanc always returns the PhotometricInterpretation is it is stored in the DICOM file stored in Orthanc whatever the Accept headers received.
• if called on ../frames/1with transfer-syntax="*", Orthanc will return the frame without any transcoding -> the PhotometricInterpretation returned in the /metadata route will match the one from the pixels
• if called with .e.g transfer-syntax="1.2.840.10008.1.2.1", Orthanc will transcode the frame and will potentially return pixels with a different PhotometricInterpretation than the one returned in /metadata

For this reason, we concluded, together with @sedghi, that using transfer-syntax="*" was the best (and only) option for OHIF to receive pixels from Orthanc with the PhotometricInterpretation matching the one returned in an earlier call to /metadata.

We are, of course, open to making Orthanc more standard compliant if it needs to be. In such a case, that would be great to have access to a few well documented test scenarios or scripts for everyone to validate their implementations.

[wayfarer3130]

I will work with @dclunie on making sure that the CP that he has started has examples of how to ask for the image in a compressed format which is standards compliant, as that is a very common question and given how many people get it wrong, it is clear we need some clarification as to what is and is not allowed.

[fedorov]

@wayfarer3130 can you let me know how we should do it with the current implementation?

Unless it is already clear to @igoroctaviano based on the discussion above.

[dclunie]

@amazy the PhotometricInterpretation question (and whether the value in the metadata response is applicable to whatever compressed or uncompressed pixel data is retrieved) should be somewhat orthogonal to the question of whether compressed pixel data byte streams are obtained in the correct manner using image/* or incorrectly using application/octet-stream.

[dclunie]

Wrt. drafting examples, many years ago (2018/10/23) I posted the following message to the WG 27 mailing list and got zero responses. It is out of date now (since we have both updated the standard, extended the use of single part responses, and added specific solutions such as AvailableTransferSyntax, which may or may not have been implemented), but I think we should be able to improve it and generalize it and come up with an exhaustive list of requests and appropriate responses:

Hi WG27 folks

I have some general questions related to compressed images that arise particularly when using anything other than baseline JPEG images sent to an origin server.

Suppose I send a multi-frame lossless JPEG 2000 (.90) image to an origin server, and for the sake of argument, assume it stores it in the supplied Transfer Syntax.

Note that it is multi-frame image, not single frame.

As a QIDO-RS and WADO-RS user agent:

1. How do I tell in what Transfer Syntax the origin server has stored an image (i.e., that I sent it as .90)?

   1a. in QIDO-RS, is (0002,0010) permitted in the request/response? 1b. in RetrieveMetadata, is (0002,0010) permitted in the response? 1c. in the dcm-parameters of the RetrieveMetadata Content-Type, even though there is no inline binary data returned?

2. What comes back in the response if I accept in a RetrieveInstance request only:

   2a. '*/*' ? 2b. 'multipart/related; type="application/dicom"' ? 2c. 'multipart/related; type="application/dicom"; transferSyntax=...90' ? 2d. 'multipart/related; type="application/dicom"; transferSyntax=...91' ? 2e. 'multipart/related; type="application/dicom"; transferSyntax=...70' ? 2f. 'multipart/related; type="application/dicom"; transferSyntax=...50' ? 2g. 'multipart/related; type="application/dicom"; transferSyntax=...1' ? 2h. 'multipart/related; type="image/jp2"' ? 2i. 'multipart/related; type="image/jpeg"' ? 2j. 'multipart/related; type="application/octet-stream"' ?

3. What comes back in the response if I accept in a RetrieveInstance request all of:

   3a. 'multipart/related; type="application/dicom"; transferSyntax=...90, multipart/related; type="application/dicom"; transferSyntax=...91, multipart/related; type="application/dicom"; transferSyntax=...70, multipart/related; type="application/dicom"; transferSyntax=...50, multipart/related; type="application/dicom"; transferSyntax=...1' ?

   3b. 'multipart/related; type="image/jp2", multipart/related; type="image/jpeg", multipart/related; type="application/octet-stream"' ?

4. What comes back in the response if I accept in a RetrieveFrame request only:

   4a. '*/*' ? 4b. 'multipart/related; type="image/jp2"' ? 4c. 'multipart/related; type="image/jpeg"' ?

5. What comes back in the response if I accept in a RetrieveRendered Frame request only:

   5a. '*/*' ? 5b. 'multipart/related; type="image/jp2"' ? 5c. 'multipart/related; type="image/jpeg"' ?

6. What is the difference, if any, if the image is sent in (.91) in the first place, rather than (.90), remembering that .91 can be lossless or lossy (reversible or irreversible)?

7. If the response contains a DICOM PS3.10 file, or its metadata, and any Transfer Syntax conversion performed involved further lossy compression:

   7a. is the SOP Instance UID changed? 7b. are any mandatory Lossy Compression attributes set?

For each answer, where do you think it is currently specified in the standard, and is the behavior required or optional?

David