DigitalSlideArchive / HistomicsTK

A Python toolkit for pathology image analysis algorithms.
https://digitalslidearchive.github.io/HistomicsTK/
Apache License 2.0
394 stars 117 forks source link

Heatmap Feature #836

Closed ghadjigeorghiou closed 4 years ago

ghadjigeorghiou commented 4 years ago

I developed a system to produce a cancer probability heatmap for tissue slides. Firstly, I extract and analyse patches from the tissue slide. Then by using a model, I assign a probability and a colour to each patch resulting in a heatmap indicating the probability of each patch containing cancerous cells. Following, using the provided methods (get_contours_from mask and get_single_annotation_document_from_contours) I translate my results into an annotation document and upload it to histomicstk. However, I face 2 issues that I am not able to resolve. The first one is the time required for the heatmap to be produced. The translation of my results to contours and then to an annotation document takes a significant amount of time. The second issue is the aesthetic of the map. Since I work with square patches the results annotation is a map of distinct squares, where I would like to be able to smooth them out.

A possible solution is to create an image of the heatmap but I couldnt find any way to upload it as an overlay over the tissue slide. Is there anything that I can do to resolve these issues?

Thanks, George

cooperlab commented 4 years ago

@ghadjigeorghiou a built-in heatmap display feature is a high priority but will have to wait until the future. There are a lot of algorithms that generate dense image-like outputs that we want to be able to display transparently over the input data. This touches on a few aspects of the project so it's not a simple task. We are wrapping up the first cycle of funding and have been focused on completing and documenting existing work.

The problems you described with using contours to render a heatmap are not easy to solve. The document you need to represent a heatmap will be very large and I can't see how to reduce this other than just optimizing the software used in processing. For the aesthetic issues you could interpolate your heatmap to a finer grid, but this will only make your first problem worse. Sorry I can't be more helpful.

kheffah commented 4 years ago

Hi @ghadjigeorghiou ,

Adding to Dr @cooperlab 's answer, if you'd like some sample code for using the contours to display a heatmap, take a look at how it's implemented in the cellularity detection superpixels code; something like this:

https://github.com/DigitalSlideArchive/HistomicsTK/blob/d26a6b8dc2892207b0d093ca91878c9cb4a00ef5/histomicstk/saliency/cellularity_detection_superpixels.py#L212

Alternatively, you can create a new "slide" offline by tiling RGB images (with heatmaps already "burned into" the image) using VIPS, and then post this to HistomicsUI for display. Something like:

https://github.com/DigitalSlideArchive/HistomicsTK/blob/rapid-review-gallery-2/histomicstk/annotations_and_masks/review_gallery.py#L388

Obviously none of these are the "ideal" approach, but thought I'd give you some starters in the mean time till an in-built heatmaps feature is added.

cooperlab commented 4 years ago

@ghadjigeorghiou I think Mohamed's solution to generate a new WSI with the heatmap embedded in the pixels is a also a good one. You just won't be able to turn it off or adjust the opacity like with an overlay.

kheffah commented 4 years ago

@ghadjigeorghiou If you decide to try out pushing the heatmap as a collection of colored annotations, be sure to read this guide on how to make this scalable so that it renders efficiently.

ghadjigeorghiou commented 4 years ago

Hi Dr @cooperlab and @kheffah ,

Sorry for not replying faster.

One of the key requirements of my system is for the user to be able to change the opacity of the map, as well as, viewing the original WSI without the heatmap being overlaid. Thus, the solution of 'burning' the heatmap into the original WSI, does not quite work for me. However, I will have a look on how to improve my current implementation based on @kheffah's suggestions and reduce the required time.

Thank you both for your fast responses and help.