NKI-AI / ahcore-old

Ahcore are the AI for Oncology histopathology core models
Apache License 2.0
0 stars 1 forks source link

Now, ROIs do not contain unmapped pixels. #12

Closed AjeyPaiK closed 1 year ago

AjeyPaiK commented 1 year ago

Fixes {#11}

Following from the issue mentioned above, I have corrected how the loss and metrics are calculated. See below for a visual explanation.

ROI_correction

This correction has led to Dice score for the background behaving as expected. After the ROI correction, both the intersection and cardinality are 0 during Dice metric computation. Whenever there is a nan in the dice, we replace it with 1.0 in ahcore. See here.

image

AjeyPaiK commented 1 year ago

@jonasteuwen @EricMarcus-ai

I think we should correct the ROIs when we measure the Dice score to avoid mistakes in metric evaluation. The regions which are unannotated may introduce ambiguity. Attaching screenshots of training curves for two experiments (ignore_index=0 in CE loss and corrected ROI in loss + metric evaluation).

image

jonasteuwen commented 1 year ago

So, after reviewing, I don't think this is the correct solution 'unmapped' pixels are actually mapped. They are background (i.e. not annotated). That inspecting the annotation gives you a different conclusion only means we need to annotate more carefully. Can't you achieve what you try to achieve by ignoring the background in the losses?

AjeyPaiK commented 1 year ago

Actually, @jonasteuwen the training logs above show exactly that. The performance indicated by grey curves is by the code in this PR. The performance indicated by blue lines is by ignore_index=0

AjeyPaiK commented 1 year ago

Agreed to setignore_index=0 during training