[Bug]: Padim doesn't support metrics AUPR

cjy513203427 commented 2 months ago

Describe the bug

When I change engine = Engine(pixel_metrics=["AUROC", "AUPR", "PRO"], image_metrics=["AUROC", "AUPR", "PRO"], task=TaskType.SEGMENTATION) to engine = Engine(pixel_metrics=["AUROC", "PRO"], image_metrics=["AUROC", "PRO"], task=TaskType.SEGMENTATION) or with default metrics Engine(task=TaskType.SEGMENTATION), it has no error reporting.

Dataset

MVTec

Model

PADiM

Steps to reproduce the behavior

Thanks for the great work!

I used API to train the code

from anomalib import TaskType
from anomalib.data import MVTec
from anomalib.engine import Engine
from anomalib.models import Padim

# configure logger
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')
logger = logging.getLogger(__name__)

datasets = ['screw', 'pill', 'capsule', 'carpet', 'grid', 'tile', 'wood', 'zipper', 'cable', 'toothbrush', 'transistor',
            'metal_nut', 'bottle', 'hazelnut', 'leather']

for dataset in datasets:
    logger.info(f"================== Processing dataset: {dataset} ==================")
    model = Padim()
    datamodule = MVTec(category=dataset, num_workers=0, train_batch_size=256,
                       eval_batch_size=256)
    # metrics is under "anomalib/metrics/"
    engine = Engine(pixel_metrics=["AUROC", "AUPR", "PRO"], image_metrics=["AUROC", "AUPR", "PRO"], task=TaskType.SEGMENTATION)

    logger.info(f"================== Start training for dataset: {dataset} ==================")
    engine.fit(model=model, datamodule=datamodule)

    logger.info(f"================== Start testing for dataset: {dataset} ==================")
    test_results = engine.test(
        model=model,
        datamodule=datamodule,
        ckpt_path=engine.trainer.checkpoint_callback.best_model_path,
    )

Then I got the error. Error message doesn't seem to be related.

What I see in the file under "src/anomalib/metrics/", It should support the following metrics:

    "AUROC",
    "AUPR",
    "AUPRO",
    "AnomalyScoreDistribution",
    "BinaryPrecisionRecallCurve",
    "F1AdaptiveThreshold",
    "F1Max",
    "F1Score",
    "ManualThreshold",
    "MinMax",
    "PRO"

Do I miss something?

OS information

OS information:

OS: [Ubuntu 22.04]
Python version: [3.10.14]
Anomalib version: [1.0.1]
PyTorch version: [2.1.2+cu121]
CUDA/cuDNN version: [12.4]
GPU models and configuration: [GeForce RTX 3090]

Expected behavior

The metrics should be supported.

Screenshots

No response

Pip/GitHub

pip

What version/branch did you use?

1.0.1

Configuration YAML

None, I used API.

Logs

/home/jinyao/anaconda3/envs/IADB/bin/python /home/jinyao/PycharmProjects/IADBE/train_test_with_padim.py 
2024-05-08 12:01:25,544 - INFO - ================== Processing dataset: screw ==================
2024-05-08 12:01:25,544 - INFO - Initializing Padim model.
2024-05-08 12:01:25,746 - INFO - Loading pretrained weights from url (https://download.pytorch.org/models/resnet18-5c106cde.pth)
2024-05-08 12:01:25,821 - INFO - ================== Start training for dataset: screw ==================
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
`Trainer(val_check_interval=1.0)` was configured so validation will run at the end of the training epoch..
2024-05-08 12:01:25,920 - INFO - Found the dataset.
You are using a CUDA device ('NVIDIA GeForce RTX 3090') that has Tensor Cores. To properly utilize them, you should set `torch.set_float32_matmul_precision('medium' | 'high')` which will trade-off precision for performance. For more details, read https://pytorch.org/docs/stable/generated/torch.set_float32_matmul_precision.html#torch.set_float32_matmul_precision
Traceback (most recent call last):
  File "/home/jinyao/PycharmProjects/IADBE/train_test_with_padim.py", line 24, in <module>
    engine.fit(model=model, datamodule=datamodule)
  File "/home/jinyao/anaconda3/envs/IADB/lib/python3.10/site-packages/anomalib/engine/engine.py", line 518, in fit
    self.trainer.fit(model, train_dataloaders, val_dataloaders, datamodule, ckpt_path)
  File "/home/jinyao/anaconda3/envs/IADB/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 544, in fit
    call._call_and_handle_interrupt(
  File "/home/jinyao/anaconda3/envs/IADB/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 44, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/jinyao/anaconda3/envs/IADB/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 580, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/home/jinyao/anaconda3/envs/IADB/lib/python3.10/site-packages/lightning/pytorch/trainer/trainer.py", line 950, in _run
    call._call_setup_hook(self)  # allow user to setup lightning_module in accelerator environment
  File "/home/jinyao/anaconda3/envs/IADB/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 93, in _call_setup_hook
    _call_callback_hooks(trainer, "setup", stage=fn)
  File "/home/jinyao/anaconda3/envs/IADB/lib/python3.10/site-packages/lightning/pytorch/trainer/call.py", line 208, in _call_callback_hooks
    fn(trainer, trainer.lightning_module, *args, **kwargs)
  File "/home/jinyao/anaconda3/envs/IADB/lib/python3.10/site-packages/anomalib/callbacks/metrics.py", line 92, in setup
    pl_module.image_metrics = create_metric_collection(image_metric_names, "image_")
  File "/home/jinyao/anaconda3/envs/IADB/lib/python3.10/site-packages/anomalib/metrics/__init__.py", line 191, in create_metric_collection
    return metric_collection_from_names(metrics, prefix)
  File "/home/jinyao/anaconda3/envs/IADB/lib/python3.10/site-packages/anomalib/metrics/__init__.py", line 60, in metric_collection_from_names
    metrics.add_metrics(metric_cls())
TypeError: PrecisionRecallCurve.__new__() missing 1 required positional argument: 'task'

Process finished with exit code 1

Code of Conduct

[x] I agree to follow this project's Code of Conduct

alexriedel1 commented 2 months ago

Please install the latest version of anomalib from source: https://github.com/openvinotoolkit/anomalib?tab=readme-ov-file#-installation

AUPR was lately updated to be a child class of BinaryPrecisionRecallCurve instead of PrecisionRecallCurve which needed a task argument

https://github.com/openvinotoolkit/anomalib/blob/f846d7a719a57af34833b0ad9760ec915e14fdbf/src/anomalib/metrics/aupr.py#L16

samet-akcay commented 2 months ago

@alexriedel1, thanks for the help!

@cjy513203427 if you want to use the pip version, we will try to release the new version on the 30th of May. Meanwhile you could use the source installation as @alexriedel1 mentioned

cjy513203427 commented 2 months ago

Please install the latest version of anomalib from source: https://github.com/openvinotoolkit/anomalib?tab=readme-ov-file#-installation

AUPR was lately updated to be a child class of BinaryPrecisionRecallCurve instead of PrecisionRecallCurve which needed a task argument

https://github.com/openvinotoolkit/anomalib/blob/f846d7a719a57af34833b0ad9760ec915e14fdbf/src/anomalib/metrics/aupr.py#L16

Thanks for the quick reply. Could you tell me how can I add a task argument for metrics if I use source mode?

samet-akcay commented 2 months ago

you should not need task anymore as the base class is binary precision recall curve

openvinotoolkit / anomalib