amazon-science / patchcore-inspection

Apache License 2.0
691 stars 142 forks source link

Bad segmentation viz #75

Open free-soellingeraj opened 1 year ago

free-soellingeraj commented 1 year ago

I am not sure if it's technically something that would be considered a bug, but it's definitely misleading. I was confused because my metric accuracy was 80-90% across the various AUC metrics, but when I viewed the array of images produced by the segmentation flag, I saw unexpected anomalies where they should not be. In my case, completely clean targets were lighting up all yellow. I added a secondary chart that used the optimal threshold on the segmentation object to print a binary mask with anomalies that exceeded the threshold. That showed that in fact, the yellow in the "good" images was not actually an anomaly, but the viz suggested otherwise.

I think that a slight tweak to that segmentation image would be desired that used a more expressive colorspace e.g. viridis to show the spectrum of values present in the segmentation object. Currently, it's just unclear because everything is yellow.

Here's an example of one "good" test case where the yellow in the segmentation is completely filtered out at the optimal threshold. In this case, left is the ground truth mask (blank because this one is good), middle is the segmentation matrix and right is the thresholded segmentation matrix.

Screen Shot 2023-06-05 at 8 34 18 PM

In my case, once I thresholded, what was revealed was that the trained model was correctly not predicting anomalies in 80% of the good cases (which agreed with the AUCs)

romanvelichkin commented 10 months ago

I've tried provided examples (bottles, cables, etc) they all shows a lot bright zones on good examples. Thank you for clarification about threshold, I'll try it.

It would be nice if you could provide some code clues how to fix threshold.