Calling viewer.setImage() multiple times can cause memory leak

[Tinanuaa]

Hi, I'm using the itk-vtk-viewer in my react app as a component on one page, and it receives live frames from websocket and then display it by setImage(image,"live-frame"). But the browser can crash if the live viewer stayed for around 5 minutes. I notice the memory and CPU usage kept going on frame received. I've tried to use setInterval to set new random images on interval to confirm the memory leak was not caused by websocket, and I tried to pass viewer.getImage("live-frame") to viewer.setImage, and this doesn't cause memory leak either. My guess is each time I pass in the new image as ndarray, which will trigger the toMultiscaleSpatialImage to call itkImageToInMemoryMultiscaleSpatialImage, and this class might somehow cache the previous frames? Or the reference to the previous image is somehow still captured somewhere?

The related code snippet is as below

import "itk-vtk-viewer";
const Monitoring = () => {
const decodeArrayBuffer = (buffer) => {
    // Get a dataview of the arraybuffer
    const view = new DataView(buffer);

    // Extract the size of the json encoded header
    // as a 32bit int in pos 0 of the blob (little endian)
    const header_size = view.getUint32(0, true);

    // Create two new ArrayBuffers for the header and image
    // by slicing the original at the appropriate positions
    const header_bytes = buffer.slice(4, header_size + 4);
    const image_bytes = buffer.slice(header_size + 4);

    // Decode the header bytestream into a string and then
    // parse into a JSON object
    const decoder = new TextDecoder("utf-8");

    // TODO: Figure out why this requires so many parsings!
    const header_json = JSON.parse(
      JSON.parse(JSON.stringify(decoder.decode(header_bytes)))
    );

    // Return a dictionary with the header and image
    return { header: header_json, image: image_bytes };
  };

const [viewer, setViewer] = React.useState(null);
const liveFrameViewer = useRef(null);
  var intervalId = setInterval(function() {
     console.log("NOTE Interval reached every 5s");
             viewer?.setImage(
        encodeArray(ndarray(new Uint16Array(2560 * 2160).map(
         () => Math.random() * 1000
        ), [ 2560,2160])),"live-frame" 
       );

// the below code doesn't cause memory leak
//viewer?.setImage(
//         viewer.getImage("live-frame"),"live-frame" 
//       );
  }, 5000);

useEffect(() => {
    if (liveFrameViewer.current) {
      console.log("NOTE create viewer")
      itkVtkViewer
        .createViewer(liveFrameViewer.current, {
          image: encodeArray(ndarray(defaultImageData, [1024, 1024])),
          use2D: true,
          imageName:"live-frame",
        })
        .then((viewer) => {
          viewer.setImageColorMap('X Ray',0);
          setViewer(viewer);
        });
    }
  }, [liveFrameViewer]);

  return (
    <Box sx={{ width: "100%", height: "100%", minHeight: "1000px" }}>
        <Grid
              id="live-viewer"
              ref={liveFrameViewer}
              sx={{
                position: "relative",
                width: "100%",
                height: "70%",
                flex: "1 1 auto",
                p:3,
                m:3
              }}
            ></Grid>

    </Box>
  );
}

[PaulHax]

Your on the right tract. Other likely memory leak sources:

• spawning of the "createImageRenderingActor"
• vtk.js volume representations

[Tinanuaa]

Hi Paul, Thanks for the confirmation, I then changed the way I'm updating the image, so I use getImage to get the current image and then just update the pyramid, scaleInfo and cachedImage to update the content of the current image (I make sure the coming image is always the same size and datatype, if not it will just call setImage directly, but I change the multiscaleSpatialImage to be only one scale so I don't need to calculate the multi scale images), then the viewer doesn't need to call itkImageToInMemoryMultiscaleSpatialImage, which solves the memory leak problem. But I just notice the viewer would be much slower in displaying the frames this way than calling setImage() directly. I guess that's why you need to use multiScaleSpatialImage in the first place, right?

[PaulHax]

Hello Tinanuaa,

Nice workaround! In the weeds =)

So slow framerate when you switch out the image? Before the switch, when it has the smaller image pyramid to use, does the rendering ever switch down to the highest resolution image/pyramid and you notice slow framerate there? There is a dropdown box with a number showing the current pyramid, highest resolution is 0.

If the framerate is low becuase the image is to large, perhaps you can use the downscaling method in InMemoryMultiscaleSpatialImage to build an image at a resonable resolution before setting scaleInfo, etc.

[Tinanuaa]

Hi Paul,

Sorry for my late reply, I was distracted to another project.

So when I updated the pyramid, I just put in the highest resolution, with only one scale( because for our images, it's always 2D and the maximum is 2560 2160 uint16 pixels, it won't be too large) and I notice the slow down happen when I was playing with some test images with the size of 256 256 uint8 pixels. I tried to just call setImage(ndarray) to let itk-vtk-viewer calculate the pyramids, it was almost realtime update. So my guess is it's not the image size slow down the display, it might be somehow the viewer wasn't triggered to update for some of the frames, which make it looks like it's slowing down. Is it possible that updating the image content and call setImage(multiscaleSpatialImage) to put back the initial image container(with new content) might not trigger a rerender?

This is just my guess, I need to play with it more to understand what happened underneath.