search

openvinotoolkit / anomalib

An anomaly detection library comprising state-of-the-art algorithms and features such as experiment management, hyper-parameter optimization, and edge inference.
https://anomalib.readthedocs.io/en/latest/
Apache License 2.0
3.43k stars 619 forks source link

[Task]: Question regarding min/max normalization computations #1535

Open j99ca opened 7 months ago

j99ca commented 7 months ago

What is the motivation for this task?

I am investigating the anomaly maps I am generating, which use the min/max normalization parameters as well as the threshold that anomalib computes during training. One thing I am trying to understand is if the min/max and threshold parameters are computed from the "best" model or the last model. The values that are being stored for min/max for the anomaly maps in the metadata.json file do not match my own analysis on the exported model. My initial thought it perhaps those values are calculated and updated every validation and not necessarily for the model that is serialized for export by anomalib (which should be based on the best observed model)? I am using the OpenVINO model (IR Format) model

I noticed there is another method for normalization, CdfNormalizationCallback. I haven't seen any configs here utilize the "cdf" value for normalization. Has anyone had any decent results using this method of normalization?

Describe the solution you'd like

To confirm the behaviour of calculating min/max parameters during training/validation

Additional context

I am using a custom dataloader for non standard image data (thermal and other image sensors), this may have muddied the results a bit but I seem to be getting a mismatch on what I expect to find for min/max on the exported model.

blaz-r commented 7 months ago

Hello. Now that I am looking at exporting, it seems like it is saving the last model, not the best one. @ashwinvaidya17 can you confirm this please? While this will not match the best model, metadata should still match the exported model, as it's taken directly from it.

Regarding the cdf normalization, I don't have any experience with it, however it is only supported by STFPM and Padim and can't be used with nncf: https://github.com/openvinotoolkit/anomalib/blob/1f50c952fc65a165884b2b94e178821dcebfbbef/src/anomalib/utils/callbacks/__init__.py#L110-L121

j99ca commented 7 months ago

I think it should use the best checkpoint? the load_model_callback for model export uses: load_model_callback = LoadModelCallback(weights_path=trainer.checkpoint_callback.best_model_path)

And the callbacks uses the ModelCheckpoint class which should support saving the best model?