search

openvinotoolkit / anomalib

An anomaly detection library comprising state-of-the-art algorithms and features such as experiment management, hyper-parameter optimization, and edge inference.
https://anomalib.readthedocs.io/en/latest/
Apache License 2.0
3.68k stars 654 forks source link

Test anomalib with unsupervised dataset #273

Closed julien-blanchon closed 2 years ago

julien-blanchon commented 2 years ago

I'm trying to adapt the current pipeline for some ad hoc dataset (ie EuroSAT) in order to test anomalib and make it more robust. For using the FolderDataset for unsupervised segmentation task.

So I'm using the EuroSAT RGB Dataset (https://madm.dfki.de/files/sentinel/EuroSAT.zip), Residential as normal and Industrial as abnormal, and of course no mask/groundtruth.

As everything is fine now on development branch with torchmetrics, I will use 24ccb585319dd10ef81778364350eb972bf1cf41.

Updating config with EuroSAT

Here is my dataset section I'm adding using to cflow, ganomaly, padim and patchcore (others settings are not modified)

dataset:
  name: eurosat
  format: folder
  path: ./datasets/EuroSAT
  normal_dir: Residential
  abnormal_dir: Industrial
  task: segmentation
  normal_test_dir: null
  mask: null
  extensions: .jpg
  split_ratio: 0.2
  seed: 0
  image_size: 64
  train_batch_size: 16
  test_batch_size: 16
  num_workers: 4
  transform_config:
    train: null
    val: null
  create_validation_set: false
  tiling:
    apply: false
    tile_size: null
    stride: null
    remove_border_count: 0
    use_random_tiling: False
    random_tile_count: 16
  inference_batch_size: 16
  fiber_batch_size: 32 #for padim

Cflow: metric pixel_AUROC not available

Epoch 0: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 344/344 [04:10<00:00,  1.38it/s]Traceback (most recent call last):                                                                                                                                              
  File "tools/train.py", line 83, in <module>
    train()
  File "tools/train.py", line 73, in train
    trainer.fit(model=model, datamodule=datamodule)
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 768, in fit
    self._call_and_handle_interrupt(
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 721, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 809, in _fit_impl
    results = self._run(model, ckpt_path=self.ckpt_path)
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1234, in _run
    results = self._run_stage()
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1321, in _run_stage
    return self._run_train()
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1351, in _run_train
    self.fit_loop.run()
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 205, in run
    self.on_advance_end()
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/fit_loop.py", line 297, in on_advance_end
    self.trainer._call_callback_hooks("on_train_epoch_end")
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1634, in _call_callback_hooks
    fn(self, self.lightning_module, *args, **kwargs)
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/callbacks/early_stopping.py", line 179, in on_train_epoch_end
    self._run_early_stopping_check(trainer)
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/callbacks/early_stopping.py", line 190, in _run_early_stopping_check
    if trainer.fast_dev_run or not self._validate_condition_metric(  # disable early_stopping with fast_dev_run
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/callbacks/early_stopping.py", line 145, in _validate_condition_metric
    raise RuntimeError(error_msg)
RuntimeError: Early stopping conditioned on metric `pixel_AUROC` which is not available. Pass in or modify your `EarlyStopping` callback to use any of the following: `image_F1Score`, `image_AUROC`

Change early_stopping > metric: pixel_AUROC to metric: image_AUROC. :heavy_check_mark:

Padim: KeyError: 'mask'

Epoch 0: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 375/375 [00:14<00:00, 25.17it/s, loss=nan]
Training took 14.995270490646362 seconds                                                                                                                                        
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]
Testing DataLoader 0:   0%|                                                                                                                             | 0/225 [00:00<?, ?it/s]Traceback (most recent call last):
  File "tools/train.py", line 83, in <module>
    train()
  File "tools/train.py", line 79, in train
    trainer.test(model=model, datamodule=datamodule)
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 936, in test
    return self._call_and_handle_interrupt(self._test_impl, model, dataloaders, ckpt_path, verbose, datamodule)
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 721, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 983, in _test_impl
    results = self._run(model, ckpt_path=self.ckpt_path)
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1234, in _run
    results = self._run_stage()
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1318, in _run_stage
    return self._run_evaluate()
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1363, in _run_evaluate
    eval_loop_results = self._evaluation_loop.run()
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 204, in run
    self.advance(*args, **kwargs)
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 153, in advance
    dl_outputs = self.epoch_loop.run(self._data_fetcher, dl_max_batches, kwargs)
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 204, in run
    self.advance(*args, **kwargs)
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 133, in advance
    self._on_evaluation_batch_end(output, **kwargs)
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/evaluation_epoch_loop.py", line 263, in _on_evaluation_batch_end
    self.trainer._call_callback_hooks(hook_name, output, *kwargs.values())
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1634, in _call_callback_hooks
    fn(self, self.lightning_module, *args, **kwargs)
  File "/home/gsd/Desktop/julien-blanchon/trash/anomalib/anomalib/utils/callbacks/visualizer_callback.py", line 141, in on_test_batch_end
    true_mask = outputs["mask"][i].cpu().numpy() * 255
KeyError: 'mask'
Testing DataLoader 0:   0%|          | 0/225 [00:00<?, ?it/s]

I get an KeyError: 'mask' after the first epoch, as ground truth is not provide the mask visualization step of on_batch_end must be disable. https://github.com/openvinotoolkit/anomalib/blob/24ccb585319dd10ef81778364350eb972bf1cf41/anomalib/utils/callbacks/visualizer_callback.py#L140 After a small fix (ie skip ground truth visualization) everything work fine. :heavy_check_mark:

#TODO: Make true_mask visualisation optional as ground thruth is not always provide 
if self.task == "segmentation" and "mask" in outputs:
   true_mask = outputs["mask"][i].cpu().numpy() * 255
   visualizer.add_image(image=true_mask, color_map="gray", title="Ground Truth")

Patchcore

Traceback (most recent call last):███████████████████████████████████████████████████████████████████████████████████████| 194/194 [01:23<00:00,  2.32it/s]
  File "tools/train.py", line 83, in <module>
    train()
  File "tools/train.py", line 73, in train
    trainer.fit(model=model, datamodule=datamodule)
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 768, in fit
    self._call_and_handle_interrupt(
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 721, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 809, in _fit_impl
    results = self._run(model, ckpt_path=self.ckpt_path)
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1234, in _run
    results = self._run_stage()
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1321, in _run_stage
    return self._run_train()
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1351, in _run_train
    self.fit_loop.run()
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 204, in run
    self.advance(*args, **kwargs)
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/fit_loop.py", line 269, in advance
    self._outputs = self.epoch_loop.run(self._data_fetcher)
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 205, in run
    self.on_advance_end()
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 255, in on_advance_end
    self._run_validation()
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 309, in _run_validation
    self.val_loop.run()
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 211, in run
    output = self.on_run_end()
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 187, in on_run_end
    self._evaluation_epoch_end(self._outputs)
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/loops/dataloader/evaluation_loop.py", line 309, in _evaluation_epoch_end
    self.trainer._call_lightning_module_hook("validation_epoch_end", output_or_outputs)
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1593, in _call_lightning_module_hook
    output = fn(*args, **kwargs)
  File "/home/gsd/Desktop/julien-blanchon/trash/anomalib/anomalib/models/components/base/anomaly_module.py", line 133, in validation_epoch_end
    self._collect_outputs(self.image_metrics, self.pixel_metrics, outputs)
  File "/home/gsd/Desktop/julien-blanchon/trash/anomalib/anomalib/models/components/base/anomaly_module.py", line 159, in _collect_outputs
    image_metric.update(output["pred_scores"], output["label"].int())
  File "/home/gsd/Desktop/julien-blanchon/trash/anomalib/anomalib/utils/metrics/collection.py", line 37, in update
    super().update(*args, **kwargs)
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/torchmetrics/collections.py", line 153, in update
    m.update(*args, **m_kwargs)
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/torchmetrics/metric.py", line 312, in wrapped_func
    update(*args, **kwargs)
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/torchmetrics/classification/stat_scores.py", line 185, in update
    tp, fp, tn, fn = _stat_scores_update(
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/torchmetrics/functional/classification/stat_scores.py", line 154, in _stat_scores_update
    preds, target, _ = _input_format_classification(
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/torchmetrics/utilities/checks.py", line 407, in _input_format_classification
    case = _check_classification_inputs(
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/torchmetrics/utilities/checks.py", line 268, in _check_classification_inputs
    case, implied_classes = _check_shape_and_type_consistency(preds, target)
  File "/home/gsd/anaconda3/envs/anomalib_env/lib/python3.8/site-packages/torchmetrics/utilities/checks.py", line 114, in _check_shape_and_type_consistency
    raise ValueError(
ValueError: Either `preds` and `target` both should have the (same) shape (N, ...), or `target` should be (N, ...) and `preds` should be (N, C, ...).
Epoch 0: 100%|██████████| 344/344 [01:44<00:00,  3.29it/s, loss=nan]

Shape error :(, any idea ? :x:

julien-blanchon commented 2 years ago

You can find an updated version here: https://github.com/julien-blanchon/anomalib And download eurosat from here: https://github.com/phelber/EuroSAT

samet-akcay commented 2 years ago

@julien-blanchon, thanks for the update.

Regarding the following error:

Shape error :(, any idea ? ❌

Patchcore currently doesn't support multiple test batch_size. See #268. If you set it to test_batch_size: 1 it will work. We'll add a hotfix in the next release.

samet-akcay commented 2 years ago

For the early stopping issue,

RuntimeError: Early stopping conditioned on metric pixel_AUROC which is not available. Pass in or modify your EarlyStopping callback to use any of the following: image_F1Score, image_AUROC

It's because you set the task: segmentation but haven't provided mask mask: null. In this case the folder dataset converts the task to classification. Maybe we should add some warning messages for this case.

julien-blanchon commented 2 years ago

Why did the folder dataset convert my task: segmentation into a classification if I don't provide any mask. Anomaly Detection is rarely supervised, most case mask aren't provide

samet-akcay commented 2 years ago

segmentation task means that there are ground-truth masks in the dataset so pixel-level performance computed. If there is no ground-truth mask in the dataset, which is quite common as you said, the task simply becomes a classification task, where the metrics are computed based on the normal/abnormal test labels.

julien-blanchon commented 2 years ago

I think I misunderstood something, most of the current segmentation model did not use the mask information for training (only for validation/visualization purpose). So why enforce the need for a ground truth map (for example here in order to make Padim work I've had to bypass the mask dependency but then everything work fine). Classification task only give a binary output as result, so we don't get any information about anomaly location (it still may be possible to sample image tile to specifically locate the abnormal part for the image)

julien-blanchon commented 2 years ago

I'm very confused, everything seems to work just fine with task: classification. So if I understand well the vocabulary, 'task' only define the evaluation metrics. -> task classification = image wide normal/abnormal (binary) prediction for evaluation (f1, auroc) -> task segmentation = pixel wide normal/abnormal (binary) prediction for evaluation (also f1, auroc)

But it didn't change the inner behaviour of my model. For a segmentation algorithm it will just average the number of abnormal pixel in order to get a image wide classification and to evaluate it as an classification model

samet-akcay commented 2 years ago

Exactly, as you said, task ensures the right performance evaluation and saving the images.

So if I understand well the vocabulary, 'task' only define the evaluation metrics. -> task classification = image wide normal/abnormal (binary) prediction for evaluation (f1, auroc) -> task segmentation = pixel wide normal/abnormal (binary) prediction for evaluation (also f1, auroc)

In addition to this, tasks saves the images accordingly supported-tasks

NOTE However, note that not every model in anomalib supports both classification and segmentation tasks. Some of them such as ganomaly, dfm, dfkde are only classification models since they do not have the mechanism to genearate the anomaly maps.

You could refer to this link to see which models support both classification and segmentation (The list has the terms detection vs segmentation, but the same idea).

Does it make any more sense?

julien-blanchon commented 2 years ago

Thanks you very much, I was struggling on that for a long time now it's crystal clear !

samet-akcay commented 2 years ago

Great! We'll try to simplify the parameters a further to make it more intuitive.