Output zip array shape not consistent with original WSI image

[Jeff-YW]

Hi there,

Amazing work and very efficient inference tool. But I do experience some issues with using the hovernext for large scale WSI image inference. I found that after post-processing which stitch the small inference tiles back together. The shape of the result "pinst_pp.zip" file has a shape that is not consistent with the original input WSI.

In this case, the inference result would not be aligned with the WSI by visualizing the selcted tile.

I wonder if there is any approach to resolve this issue?

Thanks

[eliasbaumann]

Hi @Jeff-YW

the Whole Slide dataloader is using the WSI coordinates as a basis, however the instance map is only as large as the furthest away tile from top-left. That means you should be able to zeropad to full WSI size at the respective resolution. That also means if you use coordinates, they should exactly match, as long as you are at the same resolution. The only exception here is the MIRAX format where we automatically remove the empty (defined by bounds) space of the image, so if your case is a MIRAX image, you can edit the get_shapes function in /src/post_process_utils.py to no longer adjust for the bounds.

If this does not answer you problem, can you give me a little bit more details as to which format WSI you are trying to process, and at which resolution?

[Jeff-YW]

Sure, here is a rewritten version of your message formatted for a GitHub reply:

---

Hi @eliasbaumann,

Thanks for your response!

I am using svs input and the largest resolution possible (to avoid any up-scaling or down-scaling). I modified the get_shapes function so that it matches the input shape. I used the original input width and height to create an output array.

The original code:

h, w = np.max(ds_coord, axis=0)
out_img_shape = (2, int(h + ccrop), int(w + ccrop))
out_cls_shape = (nclasses, int(h + ccrop), int(w + ccrop))

does not work for me. I am not sure if this is an issue or if it's something only happening on my side.

Any help or suggestions would be greatly appreciated!

[eliasbaumann]

I am using svs input and the largest resolution possible (to avoid any up-scaling or down-scaling). Depending on which model you are using, its running at either ~0.24mpp (Pannuke) or ~0.5mpp (Lizard) so it will produce output according to that.

E.g. here is an example if I use the pannuke trained model -> overlaying the model outputs for a ROI

import openslide
import zarr
sl = openslide.open_slide(".../TCGA-CK-4951/TCGA-CK-4951-01Z-00-DX1.abdbb15c-fd40-4a55-bf54-5668b3d4ea13.svs")
instance_map = zarr.open(".../pannuke/TCGA-CK-4951-01Z-00-DX1.abdbb15c-fd40-4a55-bf54-5668b3d4ea13/pinst_pp.zip", mode="r")
print("OpenSlide WSI dimensions:",sl.level_dimensions)
print("HoVer-NeXt results output dimensions:",instance_map.shape)

Output:

OpenSlide WSI dimensions: ((94965, 64833), (23741, 16208), (5935, 4052), (2967, 2026))
HoVer-NeXt results output dimensions: (64696, 94762)

Since openslide and numpy have a different Axis format, if you swap the numbers, the output almost matches the input.

Moreover if we now:

import numpy as np
import matplotlib.pyplot as plt
plt.figure(figsize=(16,16))
plt.imshow(np.stack([np.array(sl.read_region((30000,30000),0,(1024,1024)).convert('L')),
                        (instance_map[30000:31024,30000:31024]>0).astype(np.uint8)*200,
                        np.zeros((1024,1024),dtype=np.uint8)],
                     axis=-1)
          )

The segmentation exactly matches the images. However, for Lizard, we need to do some more work:

import cv2
# coordinates need to be halved, since the output is at 0.5mpp -> resize to full res
resized_roi = cv2.resize(instance_map[15000:15512,15000:15512],dsize=None,fx=2.0,fy=2.0, interpolation=cv2.INTER_NEAREST)

and then plugging the results into the plot:

plt.figure(figsize=(16,16))
plt.imshow(np.stack([np.array(sl.read_region((30000,30000),0,(1024,1024)).convert('L')),
                        (resized_roi>0).astype(np.uint8)*200,
                        np.zeros((1024,1024),dtype=np.uint8)],
                     axis=-1)
          )

If, for some other reason, you need the the output to exactly match the input dimensions, you can simply do this:

x,y = sl.level_dimensions[0]
out = np.zeros((y,x),dtype=np.int32)
# instance_map would need to be resized first if you are using the outputs from the lizard trained model
iy,ix = instance_map.shape
out[:iy,:ix] = instance_map

I hope I did not misunderstand what you are trying to achieve, but if so, please let me know.

[eliasbaumann]

Closing this. If this is still an issue, please let me know and re-open the issue