search

AICAN-Research / FAST-Pathology

⚡ Open-source software for deep learning-based digital pathology
BSD 2-Clause "Simplified" License
121 stars 24 forks source link

Pathologist claims the class labels are incorrect in UI #94

Open lomshabhishek opened 9 months ago

lomshabhishek commented 9 months ago

Benign is In situ and In situ should be benign Normal is invasive and invasive should be normal

I requested for a WSI where Pathologist has good confidence and showed result from FAST Pathology's BACH Classifier model. It's hard to get consensus from Pathologists, also this is from 1-2 images not a deep study.

Is it possible that either the labels are wrongly mapped or the output tensors we get from model might get sorted or reordered based on magnitude.

andreped commented 9 months ago

Hello, @lomshabhishek!

That is surprising to hear. Maybe the names that we assigned to the different classes in the BACH FPL were accidentally assigned the wrong names at some point.

Perhaps, @smistad, you could run FP on a WSI, take some snapshots, and ask one of the pathologists what they think?

Note that the BACH model was a PoC model, developed from start to finish in a few hours, to have a test model for pw-classification in FAST. So I would not necessarily trust its performance. Could just be that it mistakes one class for the other. Hard to say.

We recently published an Epithelial semantic segmentation model, which seems to work quite well. It is part of the FAST model hub, so you can download and test it through FP (or pyFAST if you'd like - I recommend starting with FP as you can easily visualize the predictions there).

smistad commented 9 months ago

I am not a pathologist, but looking at the results of one WSI, it seems to me as well that the labels are incorrect. I agree that invasive probably should be normal

Could it be that the labels are reversed? Today it is: Normal - Benign - In Situ - Invasive Maybe it should be reversed to: Invasive - In Situ - Benign - Normal? But this doesn't seem entirely correct to me other, but I am not a pathologist..

@lomshabhishek you can change the labeling yourself locally, by pressing edit pipeline and modyfing the line: Attribute classes "Normal;Benign;In Situ Carcinoma;Invasive Carcinoma"

You can then also update the colors and which channels are hidden: Attribute hidden-channels 0 Attribute channel-colors "0" "green" "1" "green" "2" "magenta" "3" "red"

andreped commented 9 months ago

I agree that invasive probably should be normal

If we observe that it is clearly something wrong, after consulting with a pathologist, we should update the BACH FPL as well, but lets first get it verified that the class names were assigned wrongly in the FPL.

But yeah, for now, I guess you can change the class names yourself in the FPL, as @smistad suggests.

smistad commented 9 months ago

Yes, we should update the pipeline in the data hub if we manage to figure out what is correct. You are the one who trained this model @andreped 🙂 Can't you check your code to see what the channel order was?

andreped commented 9 months ago

This code was from 2018 (6 years ago) and did it in one sitting. Do not have it anywhere, sadly...