most anomalous region

[DHAiRYA2048]

Hi, is it possible to extract the most anomalous region from the anomaly map? I want to just overlap the most anomalous region and not the entire anomaly map. However, I wasn't able to do it since anomaly_map is a 2D matrix so it doesn't have info about the individual contours.

Desired result; as shown in the Patchcore paper.

[tiskw]

Hi @DHAiRYA2048

Thank you for your question. First of all, as I know, the output of the PatchCore is just an anomaly map of a 2D matrix. Therefore, to get the contours to highlight high anomaly areas, we need additional techniques not written in the paper.

I think we can draw the contours by using cv2.threshold, cv2.GaussianBlur and cv2.findContours. In other words,

1. Apply cv2.threshold to an output anomaly map of the PatchCore algorithm to ignore low anomaly area,
2. Blur the thresholded anomaly map using cv2.GaussianBlur to get a smooth heat map,
3. Compute contour map using cv2.findContours.

I haven't tried it yet, but I think my idea is not perfect and some additional ideas may be necessary. I will try it when I have time (this weekend, probably), but if you have a chance to try it earlier than me, I'm happy to hear your results.

Thanks, Tetsuya

[DHAiRYA2048]

Hi, to get started, I fetched the top 20% pixels according to intensity and then drew a contour around it. I am getting acceptable results but the major issue is with the false positives (FPs) in good images. 100% of good images from the test set have FPs. To check whether my dataset needed pre-processing, I ran the model on mvtec capsule class. However, I got the same issue, anomalies are being detected as expected but the good images from the test set have random anomalies detected here and there (PFA). I got to the conclusion that thresholding top 20% pixels according to intensity is the wrong approach because it ignores the global threshold context, which is important to identify the good images (since their top 20% pixel values would be lesser than those of anomalous images).

Do you have any idea about how can one solve this?

By the way, your work is really good. I appreciate it. I was able to get it running in no time, thanks a lot for writing such a clean code.

[tiskw]

Hi @DHAiRYA2048,

Thank you for sharing your work! You seem to have succeeded in drawing the contour curve, that's great work, and the last issue may be the threshold of the anomaly score. I will describe my thoughts in the following, but let me shout out that the following is out of the scope of the PatchCore paper. I guess you've already understood, the paper evaluated their method using ROC curves that do not require determining the threshold of the anomaly score.

There are several scenarios when determining the threshold of the anomaly score, but here, we will follow MVTecAD's scenario, that is, only normal images are available in the training phase. In other words, we have to determine the threshold only from the normal images. If we have anomaly image(s), we can choose smarter approaches than the following.

The following figure is the histogram of the anomaly scores (maximum value of the anomaly map saved as a .npy file) of the training data. The distribution looks like the normal distribution, so we can assume that the anomaly score follows the normal distribution. I know this assumption is very crude, but let me go forward as it is because the identification of the distribution may require careful theoretical analysis or a large amount of numerical experiments.

Then, the threshold can be determined by a confidence interval. For example, if you use 3-sigma as the threshold, you can expect to be able to correctly classify 99.7% of normal images. The following is an example code to compute the 3 -sigma threshold.

# Train a model.
python3 main.py train -i data_train -o ./index.faiss

# Evaluate the training data.
python3 main.py predict -i data_train -o output_train

Then run the following code:

# Read NumPy file and get the anomaly scores (= maximum value of the anomaly map).
score_train = [np.max(np.load(path)) for path in pathlib.Path("output_train").glob("*.npy")]

# Compute mean/std of the anomaly scores.
score_train_mean = np.mean(score_train)
score_train_std  = np.std(score_train)

# Compute threshold.
sigma = 3
thresh = score_train_mean + sigma * score_train_std

I will write a complete code for the contour visualization including the thresholding in the next weekend or later. However, if you have a chance to do this, please share your results with me! Also, if you face another issue, please let me know again!

[tiskw]

Hi @DHAiRYA2048,

I implemented the contour map visualization. Please clone the branch feature/contour_visualization, and try the following:

# Train.
python3 main.py train -i data/mvtec_ad/capsule/train/good -o ./index.faiss

# Compute threshold.
python3 main.py thresh -i data/mvtec_ad/capsule/train/good

# Visualize contour map using the threshold value obtained by the above.
python3 main.py predict --contour 1.47 -i data/mvtec_ad/capsule/test/crack          -o output_test/crack
python3 main.py predict --contour 1.47 -i data/mvtec_ad/capsule/test/faulty_imprint -o output_test/faulty_imprint
python3 main.py predict --contour 1.47 -i data/mvtec_ad/capsule/test/good           -o output_test/good
python3 main.py predict --contour 1.47 -i data/mvtec_ad/capsule/test/poke           -o output_test/poke
python3 main.py predict --contour 1.47 -i data/mvtec_ad/capsule/test/scratch        -o output_test/scratch
python3 main.py predict --contour 1.47 -i data/mvtec_ad/capsule/test/squeeze        -o output_test/squeeze

The contour visualization will be saved under output_test directory. The following is a random sampling of the visualization:

The function to compute the threshold is auto_threshold in patchcore/utils.py, and the function for the contour visualization is PatchCore.save_anomaly_map in patchcore/patchcore.py.

Honestly, the current implementation is not the perfect and severe parameter tuning will be required. However, I hope the current implementation is a good baseline for you. If you have some additional request, feel free to let me know. Otherwise, I will merge the branch to the main.

[DHAiRYA2048]

Hey, thanks for writing the function. I was able to get good results with manual trial and error for different products using your code from the first comment. I am in the process of writing code to compute threshold as well. I think reverse engineering the formula by using the optimal threshold found manually for different products might work fine, I'll share once it gets done. The threshold depends heavily on sampling ratio (for example, sigma=3 works really well when s=0.1), so you might want to add an argument for sigma here python3 main.py thresh -i data/mvtec_ad/capsule/train/good.

[tiskw]

Hi @DHAiRYA2048,

Thank you for your feedback! Sounds right to me, so I will add the new option --sigma, will merge the branch to main, and will close this issue.