[Bug]: Efficient_AD with Openvino takes too much times to predict

[papago2355]

Describe the bug

I am using Openvino inference to predict anomalies in custom datasets. However, it takes too much times compared to other models like Padim or FastFlow. In same width/height, Padim takes 0.33 s and Fastflow takes 0.8. However, Efficient_AD takes 15s.

Is it normal that Openvino export+EfficientAD takes more times to predict compared to padim or fastflow?

Dataset

Folder

Model

Other (please specify in the field below)

Steps to reproduce the behavior

1. Export by Openvino option
2. Load Efficient_AD
3. predict anomalies

OS information

OS information:

• OS: [windows10]
• Python version: [3.8.10]
• Any other relevant information: [I'm using a custom dataset]

Expected behavior

I was expected Efficient_AD has better speed compared to Padim and Fastflow. Also, with 512x512, Efficient_AD with openvino inferencer takes 1.2s. Padim takes 0.2s.

Screenshots

No response

Pip/GitHub

pip

What version/branch did you use?

No response

Configuration YAML

dataset:
  name: belts
  root: ./datasets
  format : folder
  task: classification
  normal_dir : ok
  abnormal_dir : ng
  mask : null
  normal_test_dir : test
  extensions : ".jpg"
  train_batch_size: 1
  eval_batch_size: 16
  num_workers: 8
  image_size: 512 # dimensions to which images are resized (mandatory)
  center_crop: null # dimensions to which images are center-cropped after resizing (optional)
  normalization: none # data distribution to which the images will be normalized: [none, imagenet]
  transform_config:
    train: null
    eval: null
  test_split_mode: from_dir # options: [from_dir, synthetic]
  test_split_ratio: 0.2 # fraction of train images held out testing (usage depends on test_split_mode)
  val_split_mode: same_as_test # options: [same_as_test, from_test, synthetic]
  val_split_ratio: 0.5 # fraction of train/test images held out for validation (usage depends on val_split_mode)

model:
  name: efficientad
  teacher_out_channels: 384
  model_size: small # options: [small, medium]
  lr: 0.0001
  weight_decay: 0.00001
  padding: false
  pad_maps: true # relevant for "padding: false", see EfficientAD in lightning_model.py
  early_stopping:
    patience: 2
    metric: image_AUROC
    mode: max
  normalization_method: min_max # options: [null, min_max, cdf]

metrics:
  image:
    - F1Score
    - AUROC
  pixel:
    - F1Score
    - AUROC
  threshold:
    method: adaptive #options: [adaptive, manual]
    manual_image: null
    manual_pixel: null

visualization:
  show_images: False # show images on the screen
  save_images: False # save images to the file system
  log_images: False # log images to the available loggers (if any)
  image_save_path: null # path to which images will be saved
  mode: full # options: ["full", "simple"]

project:
  seed: 42
  path: ./results/efficient_test_model_v3

logging:
  logger: [] # options: [comet, tensorboard, wandb, csv] or combinations.
  log_graph: false # Logs the model graph to respective logger.

optimization:
  export_mode: openvino # options: torch, onnx, openvino
# PL Trainer Args. Don't add extra parameter here.
trainer:
  enable_checkpointing: true
  default_root_dir: null
  gradient_clip_val: 0
  gradient_clip_algorithm: norm
  num_nodes: 1
  devices: 1
  enable_progress_bar: true
  overfit_batches: 0.0
  track_grad_norm: -1
  check_val_every_n_epoch: 1
  fast_dev_run: false
  accumulate_grad_batches: 1
  max_epochs: 10
  min_epochs: null
  max_steps: 70000
  min_steps: null
  max_time: null
  limit_train_batches: 1.0
  limit_val_batches: 1.0
  limit_test_batches: 1.0
  limit_predict_batches: 1.0
  val_check_interval: 1.0
  log_every_n_steps: 50
  accelerator: auto # <"cpu", "gpu", "tpu", "ipu", "hpu", "auto">
  strategy: null
  sync_batchnorm: false
  precision: 32
  enable_model_summary: true
  num_sanity_val_steps: 0
  profiler: null
  benchmark: false
  deterministic: false
  reload_dataloaders_every_n_epochs: 0
  auto_lr_find: false
  replace_sampler_ddp: true
  detect_anomaly: false
  auto_scale_batch_size: false
  plugins: null
  move_metrics_to_cpu: false
  multiple_trainloader_mode: max_size_cycle

Logs

Not much logs to put in

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[blaz-r]

Hello, this certainly isn't expected behavior. Just to clarify, in your config you use 512x512 resolution which you say takes 1.2s to execute, which resolution takes 15s? In any case, EfficientAD should be fast, but it does use neural networks, so if you want the execution to be fast, you will need to run it on a GPU. I suspect this is the main culprit. Also make sure you are timing only the inference (predict time), not the model instantiation (and other parts), as that would falsely inflate the duration (unless your task involves this).

[papago2355]

Hi, thanks for help. When I use 700x1200 it takes 15s to get prediction. I only check during inference so instantiation model does not be a reason for too much time. Is it normal it takes way more times when use openvino? Other than EfficientAD, it usually takes less time to get predictions with openvino inferencer.

[papago2355]

Okay, as you mentioned, it is maybe due to use openvino in cpu option. So I tested torch_inferencer, now it is really faster. with 700x1200, I got 0.6 for prediction return. But still not sure why it works slow in openvino inferencer.

[blaz-r]

Did you also try openvino with gpu inference? You can pass device parameter and set it to GPU by using --device GPU. The reason why the CPU is so much slower (in most cases) than GPU is that neural networks can be highly parallelized, benefiting from GPU parallel execution significantly.

[samet-akcay]

@blaz-r, do you think if this is a bug or a specific use-case?

[blaz-r]

@samet-akcay I don't think this is a bug, but it surely is a bit unusual for EfficientAD to take so much time. Maybe the reason is inference on the CPU or it could also be something with OpenVINO but I can't say for sure.

[papago2355]

@samet-akcay Hi. I recentely switched inference module to TorchInferencer and it seems like the issue has been addressed. However, still does not know why it gets extremely slower when I use OpenvinoInferencer(It gives same results. No accuracy drop nor incorrect prediction. Just prediction speed gets 10x slower). Maybe it is due to huge image size?(since I am using 800x1600 image size).