Extend dicom-microscopy-viewer API to use new Codecs

[Punzo]

https://github.com/MGHComputationalPathology/dicom-microscopy-viewer

See point (5) in MGHComputationalPathology/dicom-microscopy-viewer#47

Waiting Danny finishing this project: https://github.com/cornerstonejs/codecs

This will be useful also for IDC and dcmjs in general.

Relevant IDC/dcmjs issues: https://github.com/dcmjs-org/dcmjs/issues/169 https://github.com/OHIF/Viewers/issues/2279

[hackermd]

One thing to keep in mind is that exotic image compression settings are encountered in whole slide imaging (including JPEG compression directly in RGB color space without prior color transformation, JPEG2000 compression with different subsampling factors per color channel, etc.), which are likely not encountered in radiology. We therefore have to ensure that the necessary decoder functionality of the underlying libraries (libjpeg, openjpeg, etc.) is exposed via the WASM/JS API. Maybe something to discuss with @dannyrb and @chafey.

[Punzo]

P.S.: do Danny and Chris have visibility on this repo? otherwise I think, they will not receive the notification.

[Punzo]

@hackermd @pieper I had a call with Danny and we have few options: 1) we could use https://github.com/cornerstonejs/codecs/tree/main/packages/openjpeg (J2KDecoder) to decode JPEG2000 on the client side, i.e. the images will be converted in the server into JPEG2000 before the download.

2) we could use more codecs as fallback to nativetly use more transfer syntaxes, however it would imply additional complexity and time to implement the API to use all the codecs.

3) Instead of reimplementing everything as in (2), we could use cornerstone (which is already using all the codecs) and be able to use all the transfer syntaxes. However, on the other hand, this would imply to import cornerstone in the dicom-miscroscopy-viewer and introduce a lot of complexity and it is a major effort.

I would start with (1), which is the easier/faster to implement (and should also fix https://github.com/MGHComputationalPathology/slim/issues/7). We see if it provides better performances and then we could decide in a second step to support more transfer syntaxes.

---

Example code how to use the library in (1):

// TODO: https://github.com/emscripten-core/emscripten/issues/6164
// https://emscripten.org/docs/compiling/WebAssembly.html#wasm-files-and-compilation
import regeneratorRuntime from 'regenerator-runtime';
import openjpeg from '@cornerstonejs/codec-openjpeg';
let resolveIt;
let rejectIt;
const openjpegInitialized = new Promise((resolve, reject) => {
  resolveIt = resolve;
  rejectIt = reject;
});
openjpeg.onRuntimeInitialized = async _ => {
  // Now you can use it
  resolveIt();
};
// imageFrame.pixelRepresentation === 1 <-- Signed
/**
 *
 * @param {*} compressedImageFrame
 * @param {object}  imageInfo
 * @param {boolean} imageInfo.signed -
 */
async function decodeAsync(compressedImageFrame, imageInfo) {
  // make sure openjpeg is fully initialized
  await openjpegInitialized;
  // Create a decoder instance
  const decoder = window.OpenJPEGDecoder;
  // get pointer to the source/encoded bit stream buffer in WASM memory
  // that can hold the encoded bitstream
  const encodedBufferInWASM = decoder.getEncodedBuffer(
    compressedImageFrame.length
  );
  // copy the encoded bitstream into WASM memory buffer
  encodedBufferInWASM.set(compressedImageFrame);
  // decode it
  decoder.decode();
  // get information about the decoded image
  const frameInfo = decoder.getFrameInfo();
  const interleaveMode = decoder.getInterleaveMode();
  const nearLossless = decoder.getNearLossless();
  // get the decoded pixels
  const decodedPixelsInWASM = decoder.getDecodedBuffer();
  const imageFrame = new Uint8Array(decodedPixelsInWASM.length);
  imageFrame.set(decodedPixelsInWASM);
  const encodedImageInfo = {
    columns: frameInfo.width,
    rows: frameInfo.height,
    bitsPerPixel: frameInfo.bitsPerSample,
    signed: imageInfo.signed,
    bytesPerPixel: imageInfo.bytesPerPixel,
    componentsPerPixel: frameInfo.componentCount,
  };
  // delete the instance.  Note that this frees up memory including the
  // encodedBufferInWASM and decodedPixelsInWASM invalidating them.
  // Do not use either after calling delete!
  // decoder.delete();
  const encodeOptions = {
    nearLossless,
    interleaveMode,
    frameInfo,
  };
  return {
    ...imageInfo,
    // imageFrame,
    // shim
    pixelData: imageFrame,
    // end shim
    imageInfo: encodedImageInfo,
    encodeOptions,
    ...encodeOptions,
    ...encodedImageInfo,
  };
}
export default decodeAsync;

[pieper]

My preference is to do option 1 as you suggest and minimize the dependency to just the code and not all of cornerstone.

I'm not clear on your comment about the server converting the all the images to jpeg2000 since it's only a subset of the images that are already in jpeg2000 and the others are already handled.

[Punzo]

My preference is to do option 1 as you suggest and minimize the dependency to just the code and not all of cornerstone.

ok.

I'm not clear on your comment about the server converting the all the images to jpeg2000 since it's only a subset of the images that are already in jpeg2000 and the others are already handled.

I meant that if we use only one transfer syntax in general, when we ask for the image from the server, the server will try to convert to that transfer sintax. Since we have no way to probe automatically the images type from the server, we would have to assume one transfer sintax, which could be JPEG2000, or jpeg, etc...., in case the server fail to convert the image, we could then use a different transfer sintax.

[pieper]

If I'm recalling correctly we have the option to offer multiple types in the accept header, listed in order of our preference. We don't want the server to convert from one lossy format to another so we need to be ready to accept whatever format they give. In the case of IDC, we can know in advance all the formats so it should manageable.

[hackermd]

If I'm recalling correctly we have the option to offer multiple types in the accept header, listed in order of our preference.

I prefer this approach, because it is relies on standard HTTP content negotiation, which every origin server shall support. We could further use quality values to specify our relative degree of preference.

For slide microscopy images, we should only have to deal with baseline JPEG (lossy), JPEG200 (lossless), or JPEG-LS (lossless). JPEG-LS is rarely used and should not be required for IDC. Therefore, if we tell that server that we accept media typesimage/jpeg, image/jp2, or image/jpx, we should be able to decode the content client-side using either the browser in case of image/jpeg or the cornerstonejs J2KDecoder in case of image/jp2 or image/jpx.

we could use https://github.com/cornerstonejs/codecs/tree/main/packages/openjpeg (J2KDecoder) to decode JPEG2000 on the client side, i.e. the images will be converted in the server into JPEG2000 before the download.

Why are we not just using the retrieve rendered transaction to request everything in PNG, which the browser supports natively? Is the origin server not capable of de-coding the JPEG2000 server-side?

[pieper]

Why are we not just using the retrieve rendered transaction to request everything in PNG, which the browser supports natively? Is the origin server not capable of de-coding the JPEG2000 server-side?

We should probably check performance and throughput implications

[hackermd]

Why are we not just using the retrieve rendered transaction to request everything in PNG, which the browser supports natively? Is the origin server not capable of de-coding the JPEG2000 server-side?

We should probably check performance and throughput implications

To clarify, I am suggesting this a short term solution to allow for display of JPEG 2000 compressed frames. In the mid to long term, I would prefer us to request frames in the existing transfer syntax and de-code them client side. If we cannot readily support all transfer syntaxes and would like to focus on one transfer syntax first, I would prefer JPEG-LS over JPEG 2000. Server-side encoding and client-side decoding should be significantly faster.

[Punzo]

ok, summarizing I would go like this:

1) as first option we request the image as jpeg, decode them with the browser (link), apply offscreen render, recompress in png and pass the images to OpenLayer. NOTE: this is done only for the monochrome microscopy channels, not to the RGB microscopy images. RGB microscopy images at the moment are just donwloaded as jpeg or png and passed to OpenLayer.

2) I will do perfomance tests between downloading uncompressed binary (octet-stream) vs downloading compressed (jpeg) + decoding with the browser for the monochrome microscopy channels/images.

3) if the image on the server is jpeg2000 and it could not be converted to jpeg (see https://github.com/MGHComputationalPathology/slim/issues/7): we donwload jpeg2000 (image/jp2), we use link to decode it, apply offscreen render, recompress in png and pass the images to OpenLayer. NOTE1: this would fix https://github.com/MGHComputationalPathology/slim/issues/7 too, if we apply the same logic of the monochrome microscopy channels to RGB microscopy images. Of course in the case of RGB images, we would not apply the offscreen coloring rendering step, but in future we could apply ICC color profiles correction with similarly logic of the coloring offscreen render for monochrome microscopy channels. NOTE2: I will test also if the server can convert the jpeg2000 into png, before implementing point (3).

4) depending on IDC priority, we could extend to use/decode JPEG-LS too as final step.

[hackermd]

Sounds great!

If retrieveRendered is not set, we should request frames using the Retrieve transaction using an acceptable compressed builkdata media type. I think image/jp2 is a good choice for the first codec. Moving forward, I would suggest implementing JPEG-LS as well and requesting frames with Accept: image/jp2 image/jls to allow origin servers to send frames compressed in either JPEG 2000 or JPEG-LS (which is much more performant on the level of individual frames).

If retrieveRendered is set, we should request frames using the Retrieve Rendered transaction and an acceptable rendered media type (which also includes image/jp2 and image/jls). For color images, frames are currently requested in image/png media type and are decoded client side by the browser by adding the compressed pixel data directly to img.src tag as a data URL. Moving forward, we should inject the ICC profile into the PNG iCCP header field of compressed color image frames to ensure the browser will (or at least can) display the color correctly. For monochrome images, we will need to decode frames client-side (depending on the requested media type, e.g., image/jp2 or image/jls), blend channels, re-encode them as PNG, and pass them to img.src.

[Punzo]

@hackermd @pieper done in commit https://github.com/MGHComputationalPathology/dicom-microscopy-viewer/pull/47/commits/a0279d7e37d12c2163fee592d478ed6b5e01bf4b .

I implemented the following formats (used in the given order): 1) monochorme images: A) retrieveRendered true : jp2, jpeg. (no png see inline comment at link) B) retrieveRendered false : jls, jp2, jpx, jpeg.

In all these cases the images are retrieved, decoded, colored and compressed back into png with toDataUrl. (Blending is done then in OpenLayer with the canvas operation)

2) RGB images: A) retrieveRendered true : jp2, png, jpeg. B) retrieveRendered false : jls, jp2, jpx, jpeg.

for jpeg, jls, jp2, jpx, the images are retrieved, decoded and compressed back into png with toDataUrl. for png, we just create a url with a blob (since the browser can decode this one).

TO NOTE1: I tested the decoding only for monochorme images (1) in jpeg (using the library jpegturbo). This because I don't have data in jls, jp2, jpx for testing the other two libraries (charls and openjpeg). But they have the same sintax so given proper images, they should properly work (I have already implemented all of them).

TO NOTE2: the study 1.3.6.1.4.1.5962.99.1.3802655970.225997140.1610120457442.3.0 (https://github.com/MGHComputationalPathology/slim/issues/7) is supposed to be JPEG2000 (currently uploaded in http://34.68.90.36/studies/1.3.6.1.4.1.5962.99.1.3802655970.225997140.1610120457442.3.0) , but I tried to retrieve with the following transfer syntax:

const jlsMediaType = 'image/jls';
const jlsTransferSyntaxUIDlossless = '1.2.840.10008.1.2.4.80';
const jlsTransferSyntaxUID = '1.2.840.10008.1.2.4.81';
const jp2MediaType = 'image/jp2';
const jp2TransferSyntaxUIDlossless = '1.2.840.10008.1.2.4.90';
const jp2TransferSyntaxUID = '1.2.840.10008.1.2.4.91';
const jpxMediaType = 'image/jpx';
const jpxTransferSyntaxUIDlossless = '1.2.840.10008.1.2.4.92';
const jpxTransferSyntaxUID = '1.2.840.10008.1.2.4.93';
const jpegMediaType = 'image/jpeg';
const jpegTransferSyntaxUID = '1.2.840.10008.1.2.4.50';

and none worked!, tried png as well and no success as well. Maybe is corrupted?

TO NOTE3: the google server does not convert at all between jls, jp2, jpx, jpeg formats. So if the data are in jpeg, they will only be fetched as jpeg (and so on)

[Punzo]

everything implemented in MGHComputationalPathology/dicom-microscopy-viewer#47, closing this