search

TissueImageAnalytics / tiatoolbox

Computational Pathology Toolbox developed by TIA Centre, University of Warwick.
https://warwick.ac.uk/tia
Other
375 stars 77 forks source link

Nucleus instance segmentation classification performance very poor #756

Closed deeplearningmaniac closed 6 months ago

deeplearningmaniac commented 9 months ago

Description

I've been trying to use some of the pretrained nucleus instance segmentation models to segment, in particular, epithelial cells and lymphocytes. However, from some of my testing, all of the models perform very poorly. I was hoping you might be able to help me figure out if I'm going wrong somewhere.

What I Did

I downloaded some data from the NuCLS dataset, ran the different pretrained models on some patches, and visually compared the results with the ground truth. All three models I tried -- PanNuke, MoNuSAC, and CoNSeP -- were able to segment the nuclei to varying degrees, but none of them were able to properly classify any epithelial cells, and MoNuSAC wasn't able to classify lymphocytes either.

Here is a Google Colab notebook replicating this: https://colab.research.google.com/drive/1D3h4A2R7LvMUhUbel8YullU3gkL4AVbm?usp=sharing

Am I doing something wrong? I've checked my code over and over again and don't see any obvious bugs. Any help would be appreciated.

If you'd like to try some patches other than the two that I included with the Google Drive download (which I custom uploaded for simplicity), you can find them here: https://drive.google.com/drive/folders/1eGlF9Dgu3WMEik4fqj0wJ13LKVufsfZ0 under the rgb/ and csv/ folders.

shaneahmed commented 9 months ago

Thanks @deeplearningmaniac, we will investigate this. It should work with PanNuke. There is a model trained on much larger dataset i.e., Conic that should help for now https://github.com/TissueImageAnalytics/CoNIC/blob/main/notebooks/baseline.ipynb

Taylor-Kelce commented 9 months ago

I noticed that hovernet from consep and panuke give very different classification. For example, I used sample image provided by example notebook 08 with the two different hovernets. The majority of the purple coloured nuclei are labeled as neoplastic epithelial by Pannuake: Pannuke

On the other hand, they are labeled as background with HoverNet trained from Consep: Consep.

shaneahmed commented 8 months ago

Please note that models trained on different datasets will produce different results. We do not claim that the models we have trained would be flawless but these models should be able to produce results as listed in the documentation https://tia-toolbox.readthedocs.io/en/latest/_autosummary/tiatoolbox.models.architecture.hovernet.HoVerNet.html.

The best model in our opinion, so far, is the one provided with the CONIC challenge https://github.com/TissueImageAnalytics/CoNIC/tree/main/notebooks. You can directly load the model from CONIC using the notebooks provided.

We are also further clarifying the classes for the models in #751

shaneahmed commented 8 months ago

I noticed that hovernet from consep and panuke give very different classification. For example, I used sample image provided by example notebook 08 with the two different hovernets. The majority of the purple coloured nuclei are labeled as neoplastic epithelial by Pannuake: Pannuke

On the other hand, they are labeled as background with HoverNet trained from Consep: Consep.

The classification labels might be incorrect in this case. We are adding the documentation in #751