PictureInPictureViewer displays square black area outside the OME-TIFF image

[andreasg123]

Before attempting to address this issue on my own, I would like to ask if this is already a known issue. When viewing an OME-TIFF 32361 x 11636 image tile pyramid, the image is extended to a square area with a black background. When zooming out, everything outside that square area is transparent such that the web page background (white) is shown. I would like the black area to be transparent, too.

Here is some of the tiffinfo output in case it makes a difference.

TIFF Directory at offset 0x1fc14e27 (532762151)
  Image Width: 32361 Image Length: 11636
  Tile Width: 1024 Tile Length: 1024
  Bits/Sample: 8
  Sample Format: unsigned integer
  Compression Scheme: AdobeDeflate
  Photometric Interpretation: min-is-black
  Samples/Pixel: 1
  Planar Configuration: single image plane
  SubIFD Offsets: 532749233 532751087 532751789 532752235 532752585 532752903 532753221
  Software: OME Bio-Formats 6.4.0
TIFF Directory at offset 0x1fc318c5 (532879557)
  Image Width: 32361 Image Length: 11636
  Tile Width: 1024 Tile Length: 1024
  Bits/Sample: 8
  Sample Format: unsigned integer
  Compression Scheme: AdobeDeflate
  Photometric Interpretation: min-is-black
  Samples/Pixel: 1
  Planar Configuration: single image plane
  SubIFD Offsets: 532753539 532755393 532756095 532756541 532756891 532757209 532757527
  Software: OME Bio-Formats 6.4.0
TIFF Directory at offset 0x1fc3323b (532886075)
  Image Width: 32361 Image Length: 11636
  Tile Width: 1024 Tile Length: 1024
  Bits/Sample: 8
  Sample Format: unsigned integer
  Compression Scheme: AdobeDeflate
  Photometric Interpretation: min-is-black
  Samples/Pixel: 1
  Planar Configuration: single image plane
  SubIFD Offsets: 532757845 532759699 532760401 532760847 532761197 532761515 532761833
  Software: OME Bio-Formats 6.4.0

[ilan-gold]

@andreasg123 I will try to reproduce this on my own but in the meantime can you provide a code snippet or the like? It's possible my recent PR #301 solved this but when I try changing the height/width manually in Avivator.js it seems like the PIP box moves correctly:

[ilan-gold]

I checkout the branch from 0.7.0 release and got a similar result. I will try reproducing in a clean project now.

[ilan-gold]

Also, can you send a screenshot of what happens when you try zooming in?

[ilan-gold]

Lastly, it looks like your image is a RGB image of some sort. We have support coming for rendering those properly from interleaved data. If you want me to take a look at why your data isn't displaying correctly I can do that too. Let me know if you can send me the image somehow.

[andreasg123]

@ilan-gold, here is the code that I'm using (still very minimal):

import { PictureInPictureViewer, createOMETiffLoader } from "@hms-dbmi/viv";
import { FunctionComponent, useEffect, useLayoutEffect, useRef, useState } from "react";
import { makeStyles } from "@material-ui/core/styles";

const useStyles = makeStyles(() => ({
  root: {
    position: "relative",
    marginTop: "4px",
    flex: "1 1 auto"
  }
}));

type ImageViewerProps = {
}

const ImageViewer: FunctionComponent<ImageViewerProps> = () => {
  const classes = useStyles();
  const url = "/images/20190219_Slide_3.ome.tiff";
  //const url = "/images/HandEuncompressed_Scan1.ome.tif";
  const targetRef = useRef<HTMLDivElement>();
  const [size, setSize] = useState({width: 0, height: 0});
  const [loader, setLoader] = useState();
  useEffect(() => {
    async function load() {
      // See here for information about offsets: http://viv.gehlenborglab.org/#data-preparation
      const res = await fetch(url.replace(/ome\.tif(f?)/gi, "offsets.json"));
      console.log(res);
      const isOffsets404 = res.status === 404;
      const offsets = !isOffsets404 ? await res.json() : [];
      setLoader(await createOMETiffLoader({ urlOrFile: url, offsets, headers: {} }));
    }
    load();
  }, [url]);
  useLayoutEffect(() => {
    if (targetRef.current) {
      setSize({
        width: targetRef.current.offsetWidth,
        height: targetRef.current.offsetHeight
      });
    }
  }, []);
  const sliders = [[0,255], [0,255], [0,255]];
  const colors = [[255, 0, 0], [0, 255, 0], [0, 0, 255]];
  const isOn = [true, true, true];
  const selections = [{ channel: 1 }, { channel: 2 }];
  const initialViewState = {
    zoom: -5,
    target: [15000, 5000, 0],
    width: size.width,
    height: size.height
  };
  const colormap = "";
  const overview = {
    boundingBoxColor: [0, 0, 255],
    position: "bottom-left"
  };
  console.log("loader", loader);
  const overviewOn = true;
  const viewer = loader && size.width ? (
    <PictureInPictureViewer

      loader={loader}
      sliderValues={sliders}
      colorValues={colors}
      channelIsOn={isOn}
      loaderSelection={selections}
      initialViewState={initialViewState}
      colormap={colormap.length > 0 && colormap}
      overview={overview}
      overviewOn={overviewOn}
      lensSelection={0}
    />
  ) : ( <></> );
  return (
    <div ref={targetRef} className={classes.root}>
      {viewer}
    </div>
  );
}

export default ImageViewer;

Regarding the image, I would have to check if it is proprietary. It is 8-bit RGB.

[andreasg123]

Here is a zoomed-in screenshot. It does exactly what it should. I should note that the viewer always uses the rest of the web page so that the placement of overview and scale is correct. I'm only having issues with the black area, something that you wouldn't notice in Avivator because it uses a black background.

[ilan-gold]

@andreasg123 Nothing jumps out at me about the code, except perhaps your const selections = [{ channel: 1 }, { channel: 2 }]; which should probably be const selections = [{ channel: 0 }, { channel: 1 }, { channel: 2 }];

Is your issue that you do not want to make the background black? Fixing the channel selections might help as well. I will look into the issue of the overly-aggressive padding in the meantime.

I made a code sandbox - https://codesandbox.io/s/snowy-meadow-ndk0x?file=/src/App.js in case you want to mess around - I do notice what you are saying, but my solution would be "make the background black" for the time being. It is not uncommon in other viewers to have a black (see here for what QPath does). Maybe it would be worth wrapping the component in a div by default that has a black background?

[ilan-gold]

@andreasg123 Here is Viv "out in the wild" with a black background for an RGB image: https://portal.hubmapconsortium.org/browse/dataset/02372aa02897532a31d0100079a99aeb

[andreasg123]

Thanks for looking into it. I'll consider making the background black if that's the best workaround. I'm not completely opposed to having the black padding rounded up to the tile size. However, I would prefer the vertical padding to stop at the next tile size (32 x 12 tiles) instead of having an area of 32 x 32 tiles.

[andreasg123]

Your example has the same issue when making the background white.

[ilan-gold]

Huh, that is weird - can you provide your browser, its version, and your OS? I do not get that same issue:

[andreasg123]

I used DevTools to make the background white to illustrate the issue.

macOS 10.15.7, Chrome 87.0.4280.67

[ilan-gold]

@andreasg123 I am going to resolve this "once and for all" - the PR #302 obviates the need for padding tiles so we will stop doing that.

[ilan-gold]

@andreasg123 This should be resolved in 0.8.0

[andreasg123]

@ilan-gold, yes that works for me. Thanks for the fix.

[ilan-gold]

@andreasg123 No problem. Happy to help.

[andreasg123]

@ilan-gold, maybe you would be interested to add this code snippet to your examples that updates the viewer size (size.width and size.height are passed to the viewer):

  const targetRef = useRef<HTMLDivElement>();
  const [size, setSize] = useState({width: 0, height: 0});
  const update_size = () => {
    if (targetRef.current) {
      setSize({
        width: targetRef.current.offsetWidth,
        height: targetRef.current.offsetHeight
      });
    }
  };
  let timer = null;
  useEffect(() => {
    const handle_resize = () => {
      clearTimeout(timer);
      timer = setTimeout(update_size, 100);
    };
    window.addEventListener("resize", handle_resize);
    update_size();
    return () => {
      window.removeEventListener("resize", handle_resize);
    };
  }, []);
  return (
    <div ref={targetRef}>

[ilan-gold]

@andreasg123 That is a good idea. I wonder if this is something that should wrap our components by default. Do you think people would be interested in that? I will look into doing that or will add this to our docs. Thanks!

[andreasg123]

@ilan-gold, it probably would be nice if you added it to Avivator. I wouldn't wrap the base component because it would make it less flexible for other people's use.

You may also be interested in this code snippet for the initial zoom (to fit the image inside the browser window):

  const zoom = !isLoading && Math.log2(Math.min(size.width / loader.width, size.height / loader.height));
  const initialViewState = {
    zoom,
    target: [0.5 * loader.width, 0.5 * loader.height, 0],
  };

[ilan-gold]

@andreasg123 I am not sure why the docs look like this (the function takes two arguments loader, {height, width} not four arguments) but we have a function already that should be able to be imported in your project (which is not well advertised and apparently marked as "internal"): http://viv.gehlenborglab.org/#getdefaultinitialviewstate

I will fix the docs for this.

[andreasg123]

@ilan-gold, I previously found that function. I guess I'm just not agreeing with what it does. With a pyramid, getDefaultInitialViewState returns the nearest fitting integer zoom. That can result in dimensions that are up to a factor of 2 smaller than the browser window. Instead, my suggestion returns the floating point zoom that really makes the image fit into the browser window. I'm not sure why you restricted yourself to an integer zoom because the mouse scroll wheel needs about 130 clicks to go from one zoom level to the next so that there are many floating point zoom values in between.

[ilan-gold]

@andreasg123 I see what you mean. I had no reason for doing it this way other than I did it in a bit of a haste. I'll make a PR for this - thanks for the suggestion.

[ilan-gold]

I may add some back-off though on the zoom - filling the screen seems like it might be a little too much.