search

openvinotoolkit / anomalib

An anomaly detection library comprising state-of-the-art algorithms and features such as experiment management, hyper-parameter optimization, and edge inference.
https://anomalib.readthedocs.io/en/latest/
Apache License 2.0
3.82k stars 679 forks source link

[Bug]: When computing AU-PRO got value 1.21 #1651

Open cugwu opened 10 months ago

cugwu commented 10 months ago

Describe the bug

I'm using FastFlow with wide_resnet50_2. I'm training FastFlow on MvTec. For the "carpet" class I get an AU-PRO of 1.21, which is impossible. This also happened with the same configuration of FastFlow for the class "capsule" and "grid". There was a first tentative that went well but when I ran again the code I got AU-PRO of 1.21

Dataset

MVTec

Model

FastFlow

Steps to reproduce the behavior

  1. Install anomalib
  2. pip install anomalib, in your python enviroment
  3. change config.yaml of FastFlow: dataset>category>carpet and model>backbone>wide_resnet50_2

OS information

OS information:

Expected behavior

I'm expecting that the maximum value od AUPRO is 1.00

Screenshots

Screenshot 2024-01-18 at 10 42 43

Pip/GitHub

pip

What version/branch did you use?

I used the following command: pip install anomalib

Configuration YAML

dataset:
  name: mvtec
  format: mvtec
  path: /data/users/cugwu/ad_data/mvtec
  task: segmentation
  category: carpet
  train_batch_size: 32
  eval_batch_size: 32
  num_workers: 2
  image_size: 256 # dimensions to which images are resized (mandatory) options: [256, 256, 448, 384]
  center_crop: null # dimensions to which images are center-cropped after resizing (optional)
  normalization: imagenet # data distribution to which the images will be normalized: [none, imagenet]
  transform_config:
    train: null
    eval: null
  test_split_mode: from_dir # options: [from_dir, synthetic]
  test_split_ratio: 0.2 # fraction of train images held out testing (usage depends on test_split_mode)
  val_split_mode: same_as_test # options: [same_as_test, from_test, synthetic]
  val_split_ratio: 0.5 # fraction of train/test images held out for validation (usage depends on val_split_mode)
  tiling:
    apply: false
    tile_size: null
    stride: null
    remove_border_count: 0
    use_random_tiling: False
    random_tile_count: 16

model:
  name: fastflow
  backbone: wide_resnet50_2 # options: [resnet18, wide_resnet50_2, cait_m48_448, deit_base_distilled_patch16_384]
  pre_trained: true
  flow_steps: 8 # options: [8, 8, 20, 20] - for each supported backbone
  hidden_ratio: 1.0 # options: [1.0, 1.0, 0.16, 0.16] - for each supported backbone
  conv3x3_only: True # options: [True, False, False, False] - for each supported backbone
  lr: 0.001
  weight_decay: 0.00001
  early_stopping:
    patience: 3
    metric: pixel_AUROC
    mode: max
  normalization_method: min_max # options: [null, min_max, cdf]

metrics:
  image:
    - AUROC
  pixel:
    - AUROC
    - AUPRO
  threshold:
    method: adaptive #options: [adaptive, manual]
    manual_image: null
    manual_pixel: null

visualization:
  show_images: False # show images on the screen
  save_images: True # save images to the file system
  log_images: True # log images to the available loggers (if any)
  image_save_path: null # path to which images will be saved
  mode: full # options: ["full", "simple"]

project:
  seed: 42
  path: /data/users/cugwu/ad_data/results/F4AD/anomalib/IN

logging:
  logger: [] # options: [comet, tensorboard, wandb, csv] or combinations.
  log_graph: false # Logs the model graph to respective logger.

optimization:
  export_mode: null # options: torch, onnx, openvino

# PL Trainer Args. Don't add extra parameter here.
trainer:
  enable_checkpointing: true
  default_root_dir: null
  gradient_clip_val: 0
  gradient_clip_algorithm: norm
  num_nodes: 1
  devices: 1
  enable_progress_bar: true
  overfit_batches: 0.0
  track_grad_norm: -1
  check_val_every_n_epoch: 1 # Don't validate before extracting features.
  fast_dev_run: false
  accumulate_grad_batches: 1
  max_epochs: 500
  min_epochs: null
  max_steps: -1
  min_steps: null
  max_time: null
  limit_train_batches: 1.0
  limit_val_batches: 1.0
  limit_test_batches: 1.0
  limit_predict_batches: 1.0
  val_check_interval: 1.0 # Don't validate before extracting features.
  log_every_n_steps: 50
  accelerator: auto # <"cpu", "gpu", "tpu", "ipu", "hpu", "auto">
  strategy: null
  sync_batchnorm: false
  precision: 32
  enable_model_summary: true
  num_sanity_val_steps: 0
  profiler: null
  benchmark: false
  deterministic: false
  reload_dataloaders_every_n_epochs: 0
  auto_lr_find: false
  replace_sampler_ddp: true
  detect_anomaly: false
  auto_scale_batch_size: false
  plugins: null
  move_metrics_to_cpu: false
  multiple_trainloader_mode: max_size_cycle

Logs

Training: 0it [00:00, ?it/s]
Training:   0%|          | 0/13 [00:00<?, ?it/s]
Epoch 0:   0%|          | 0/13 [00:00<?, ?it/s] 
Epoch 0:   8%|▊         | 1/13 [00:05<01:01,  5.09s/it]
Epoch 0:   8%|▊         | 1/13 [00:05<01:01,  5.14s/it, loss=9.24e+05, train_loss_step=9.24e+5]
Epoch 0:  15%|█▌        | 2/13 [00:05<00:29,  2.65s/it, loss=9.24e+05, train_loss_step=9.24e+5]
Epoch 0:  15%|█▌        | 2/13 [00:05<00:29,  2.71s/it, loss=1.64e+11, train_loss_step=3.29e+11]
Epoch 0:  23%|██▎       | 3/13 [00:05<00:18,  1.89s/it, loss=1.64e+11, train_loss_step=3.29e+11]
Epoch 0:  23%|██▎       | 3/13 [00:05<00:19,  1.92s/it, loss=1.1e+11, train_loss_step=2.27e+5]  
Epoch 0:  31%|███       | 4/13 [00:05<00:13,  1.48s/it, loss=1.1e+11, train_loss_step=2.27e+5]
Epoch 0:  31%|███       | 4/13 [00:06<00:13,  1.50s/it, loss=8.22e+10, train_loss_step=1.79e+5]
Epoch 0:  38%|███▊      | 5/13 [00:07<00:12,  1.55s/it, loss=8.22e+10, train_loss_step=1.79e+5]
Epoch 0:  38%|███▊      | 5/13 [00:07<00:12,  1.57s/it, loss=6.58e+10, train_loss_step=2.4e+5] 
Epoch 0:  46%|████▌     | 6/13 [00:08<00:09,  1.34s/it, loss=6.58e+10, train_loss_step=2.4e+5]
Epoch 0:  46%|████▌     | 6/13 [00:08<00:09,  1.35s/it, loss=5.48e+10, train_loss_step=1.9e+5]
Epoch 0:  54%|█████▍    | 7/13 [00:09<00:08,  1.41s/it, loss=5.48e+10, train_loss_step=1.9e+5]
Epoch 0:  54%|█████▍    | 7/13 [00:09<00:08,  1.42s/it, loss=4.7e+10, train_loss_step=7.6e+4] 
Epoch 0:  62%|██████▏   | 8/13 [00:10<00:06,  1.26s/it, loss=4.7e+10, train_loss_step=7.6e+4]
Epoch 0:  62%|██████▏   | 8/13 [00:10<00:06,  1.27s/it, loss=4.11e+10, train_loss_step=-5.42e+4]
Epoch 0:  69%|██████▉   | 9/13 [00:10<00:04,  1.20s/it, loss=4.11e+10, train_loss_step=-5.42e+4]
Epoch 0:  69%|██████▉   | 9/13 [00:10<00:04,  1.21s/it, loss=3.65e+10, train_loss_step=-1.94e+5]

Validation: 0it [00:00, ?it/s]

Validation:   0%|          | 0/4 [00:00<?, ?it/s]

Validation DataLoader 0:   0%|          | 0/4 [00:00<?, ?it/s]

Validation DataLoader 0:  25%|██▌       | 1/4 [00:00<00:00,  6.53it/s]
Epoch 0:  77%|███████▋  | 10/13 [00:13<00:04,  1.40s/it, loss=3.65e+10, train_loss_step=-1.94e+5]

Validation DataLoader 0:  50%|█████     | 2/4 [00:00<00:00,  5.93it/s]
Epoch 0:  85%|████████▍ | 11/13 [00:14<00:02,  1.29s/it, loss=3.65e+10, train_loss_step=-1.94e+5]

Validation DataLoader 0:  75%|███████▌  | 3/4 [00:01<00:00,  1.66it/s]
Epoch 0:  92%|█████████▏| 12/13 [00:15<00:01,  1.30s/it, loss=3.65e+10, train_loss_step=-1.94e+5]

Validation DataLoader 0: 100%|██████████| 4/4 [00:01<00:00,  2.11it/s]
Epoch 0: 100%|██████████| 13/13 [00:15<00:00,  1.21s/it, loss=3.65e+10, train_loss_step=-1.94e+5]
Epoch 0: 100%|██████████| 13/13 [00:17<00:00,  1.37s/it, loss=3.65e+10, train_loss_step=-1.94e+5, pixel_AUROC=0.978, pixel_AUPRO=0.929]

                                                                      
Epoch 0: 100%|██████████| 13/13 [00:17<00:00,  1.37s/it, loss=3.65e+10, train_loss_step=-1.94e+5, pixel_AUROC=0.978, pixel_AUPRO=0.929, train_loss_epoch=3.76e+10]
Epoch 0:   0%|          | 0/13 [00:00<?, ?it/s, loss=3.65e+10, train_loss_step=-1.94e+5, pixel_AUROC=0.978, pixel_AUPRO=0.929, train_loss_epoch=3.76e+10]         
Epoch 1:   0%|          | 0/13 [00:00<?, ?it/s, loss=3.65e+10, train_loss_step=-1.94e+5, pixel_AUROC=0.978, pixel_AUPRO=0.929, train_loss_epoch=3.76e+10]
Epoch 1:   8%|▊         | 1/13 [00:02<00:35,  2.92s/it, loss=3.65e+10, train_loss_step=-1.94e+5, pixel_AUROC=0.978, pixel_AUPRO=0.929, train_loss_epoch=3.76e+10]
Epoch 1:   8%|▊         | 1/13 [00:03<00:36,  3.02s/it, loss=3.29e+10, train_loss_step=-2.7e+5, pixel_AUROC=0.978, pixel_AUPRO=0.929, train_loss_epoch=3.76e+10] 
Epoch 1:  15%|█▌        | 2/13 [00:03<00:17,  1.60s/it, loss=3.29e+10, train_loss_step=-2.7e+5, pixel_AUROC=0.978, pixel_AUPRO=0.929, train_loss_epoch=3.76e+10]
Epoch 1:  15%|█▌        | 2/13 [00:03<00:18,  1.65s/it, loss=2.99e+10, train_loss_step=-3.45e+5, pixel_AUROC=0.978, pixel_AUPRO=0.929, train_loss_epoch=3.76e+10]
Epoch 1:  23%|██▎       | 3/13 [00:04<00:16,  1.65s/it, loss=2.99e+10, train_loss_step=-3.45e+5, pixel_AUROC=0.978, pixel_AUPRO=0.929, train_loss_epoch=3.76e+10]
Epoch 1:  23%|██▎       | 3/13 [00:05<00:16,  1.68s/it, loss=2.74e+10, train_loss_step=-4.24e+5, pixel_AUROC=0.978, pixel_AUPRO=0.929, train_loss_epoch=3.76e+10]
Epoch 1:  31%|███       | 4/13 [00:05<00:11,  1.30s/it, loss=2.74e+10, train_loss_step=-4.24e+5, pixel_AUROC=0.978, pixel_AUPRO=0.929, train_loss_epoch=3.76e+10]
Epoch 1:  31%|███       | 4/13 [00:05<00:11,  1.32s/it, loss=2.53e+10, train_loss_step=-4.64e+5, pixel_AUROC=0.978, pixel_AUPRO=0.929, train_loss_epoch=3.76e+10]
Epoch 1:  38%|███▊      | 5/13 [00:06<00:11,  1.40s/it, loss=2.53e+10, train_loss_step=-4.64e+5, pixel_AUROC=0.978, pixel_AUPRO=0.929, train_loss_epoch=3.76e+10]
Epoch 1:  38%|███▊      | 5/13 [00:07<00:11,  1.42s/it, loss=2.35e+10, train_loss_step=-5.44e+5, pixel_AUROC=0.978, pixel_AUPRO=0.929, train_loss_epoch=3.76e+10]
Epoch 1:  46%|████▌     | 6/13 [00:07<00:08,  1.20s/it, loss=2.35e+10, train_loss_step=-5.44e+5, pixel_AUROC=0.978, pixel_AUPRO=0.929, train_loss_epoch=3.76e+10]
Epoch 1:  46%|████▌     | 6/13 [00:07<00:08,  1.22s/it, loss=2.19e+10, train_loss_step=-5.77e+5, pixel_AUROC=0.978, pixel_AUPRO=0.929, train_loss_epoch=3.76e+10]
Epoch 1:  54%|█████▍    | 7/13 [00:09<00:07,  1.29s/it, loss=2.19e+10, train_loss_step=-5.77e+5, pixel_AUROC=0.978, pixel_AUPRO=0.929, train_loss_epoch=3.76e+10]
Epoch 1:  54%|█████▍    | 7/13 [00:09<00:07,  1.31s/it, loss=2.06e+10, train_loss_step=-6.16e+5, pixel_AUROC=0.978, pixel_AUPRO=0.929, train_loss_epoch=3.76e+10]
Epoch 1:  62%|██████▏   | 8/13 [00:09<00:05,  1.16s/it, loss=2.06e+10, train_loss_step=-6.16e+5, pixel_AUROC=0.978, pixel_AUPRO=0.929, train_loss_epoch=3.76e+10]
Epoch 1:  62%|██████▏   | 8/13 [00:09<00:05,  1.17s/it, loss=1.93e+10, train_loss_step=-6.61e+5, pixel_AUROC=0.978, pixel_AUPRO=0.929, train_loss_epoch=3.76e+10]
Epoch 1:  69%|██████▉   | 9/13 [00:09<00:04,  1.11s/it, loss=1.93e+10, train_loss_step=-6.61e+5, pixel_AUROC=0.978, pixel_AUPRO=0.929, train_loss_epoch=3.76e+10]
Epoch 1:  69%|██████▉   | 9/13 [00:10<00:04,  1.11s/it, loss=1.83e+10, train_loss_step=-7.14e+5, pixel_AUROC=0.978, pixel_AUPRO=0.929, train_loss_epoch=3.76e+10]

Validation: 0it [00:00, ?it/s]

Validation:   0%|          | 0/4 [00:00<?, ?it/s]

Validation DataLoader 0:   0%|          | 0/4 [00:00<?, ?it/s]

Validation DataLoader 0:  25%|██▌       | 1/4 [00:00<00:00,  7.21it/s]
Epoch 1:  77%|███████▋  | 10/13 [00:12<00:03,  1.29s/it, loss=1.83e+10, train_loss_step=-7.14e+5, pixel_AUROC=0.978, pixel_AUPRO=0.929, train_loss_epoch=3.76e+10]

Validation DataLoader 0:  50%|█████     | 2/4 [00:00<00:00,  6.30it/s]
Epoch 1:  85%|████████▍ | 11/13 [00:13<00:02,  1.19s/it, loss=1.83e+10, train_loss_step=-7.14e+5, pixel_AUROC=0.978, pixel_AUPRO=0.929, train_loss_epoch=3.76e+10]

Validation DataLoader 0:  75%|███████▌  | 3/4 [00:01<00:00,  1.68it/s]
Epoch 1:  92%|█████████▏| 12/13 [00:14<00:01,  1.21s/it, loss=1.83e+10, train_loss_step=-7.14e+5, pixel_AUROC=0.978, pixel_AUPRO=0.929, train_loss_epoch=3.76e+10]

Validation DataLoader 0: 100%|██████████| 4/4 [00:01<00:00,  2.15it/s]
Epoch 1: 100%|██████████| 13/13 [00:14<00:00,  1.12s/it, loss=1.83e+10, train_loss_step=-7.14e+5, pixel_AUROC=0.978, pixel_AUPRO=0.929, train_loss_epoch=3.76e+10]
Epoch 1: 100%|██████████| 13/13 [00:16<00:00,  1.28s/it, loss=1.83e+10, train_loss_step=-7.14e+5, pixel_AUROC=0.975, pixel_AUPRO=0.905, train_loss_epoch=3.76e+10]

                                                                      
Epoch 1: 100%|██████████| 13/13 [00:16<00:00,  1.28s/it, loss=1.83e+10, train_loss_step=-7.14e+5, pixel_AUROC=0.975, pixel_AUPRO=0.905, train_loss_epoch=-5.07e+5]
Epoch 1:   0%|          | 0/13 [00:00<?, ?it/s, loss=1.83e+10, train_loss_step=-7.14e+5, pixel_AUROC=0.975, pixel_AUPRO=0.905, train_loss_epoch=-5.07e+5]         
Epoch 2:   0%|          | 0/13 [00:00<?, ?it/s, loss=1.83e+10, train_loss_step=-7.14e+5, pixel_AUROC=0.975, pixel_AUPRO=0.905, train_loss_epoch=-5.07e+5]
Epoch 2:   8%|▊         | 1/13 [00:03<00:37,  3.16s/it, loss=1.83e+10, train_loss_step=-7.14e+5, pixel_AUROC=0.975, pixel_AUPRO=0.905, train_loss_epoch=-5.07e+5]
Epoch 2:   8%|▊         | 1/13 [00:03<00:39,  3.26s/it, loss=1.73e+10, train_loss_step=-6.84e+5, pixel_AUROC=0.975, pixel_AUPRO=0.905, train_loss_epoch=-5.07e+5]
Epoch 2:  15%|█▌        | 2/13 [00:03<00:18,  1.72s/it, loss=1.73e+10, train_loss_step=-6.84e+5, pixel_AUROC=0.975, pixel_AUPRO=0.905, train_loss_epoch=-5.07e+5]
Epoch 2:  15%|█▌        | 2/13 [00:03<00:19,  1.77s/it, loss=1.64e+10, train_loss_step=-7.16e+5, pixel_AUROC=0.975, pixel_AUPRO=0.905, train_loss_epoch=-5.07e+5]
Epoch 2:  23%|██▎       | 3/13 [00:05<00:17,  1.74s/it, loss=1.64e+10, train_loss_step=-7.16e+5, pixel_AUROC=0.975, pixel_AUPRO=0.905, train_loss_epoch=-5.07e+5]
Epoch 2:  23%|██▎       | 3/13 [00:05<00:17,  1.78s/it, loss=1.64e+10, train_loss_step=-7.33e+5, pixel_AUROC=0.975, pixel_AUPRO=0.905, train_loss_epoch=-5.07e+5]
Epoch 2:  31%|███       | 4/13 [00:05<00:12,  1.37s/it, loss=1.64e+10, train_loss_step=-7.33e+5, pixel_AUROC=0.975, pixel_AUPRO=0.905, train_loss_epoch=-5.07e+5]
Epoch 2:  31%|███       | 4/13 [00:05<00:12,  1.40s/it, loss=-3.41e+05, train_loss_step=-7.33e+5, pixel_AUROC=0.975, pixel_AUPRO=0.905, train_loss_epoch=-5.07e+5]
Epoch 2:  38%|███▊      | 5/13 [00:07<00:11,  1.45s/it, loss=-3.41e+05, train_loss_step=-7.33e+5, pixel_AUROC=0.975, pixel_AUPRO=0.905, train_loss_epoch=-5.07e+5]
Epoch 2:  38%|███▊      | 5/13 [00:07<00:11,  1.47s/it, loss=-3.93e+05, train_loss_step=-8.17e+5, pixel_AUROC=0.975, pixel_AUPRO=0.905, train_loss_epoch=-5.07e+5]
Epoch 2:  46%|████▌     | 6/13 [00:07<00:08,  1.25s/it, loss=-3.93e+05, train_loss_step=-8.17e+5, pixel_AUROC=0.975, pixel_AUPRO=0.905, train_loss_epoch=-5.07e+5]
Epoch 2:  46%|████▌     | 6/13 [00:07<00:08,  1.27s/it, loss=-4.44e+05, train_loss_step=-8.34e+5, pixel_AUROC=0.975, pixel_AUPRO=0.905, train_loss_epoch=-5.07e+5]
Epoch 2:  54%|█████▍    | 7/13 [00:09<00:08,  1.33s/it, loss=-4.44e+05, train_loss_step=-8.34e+5, pixel_AUROC=0.975, pixel_AUPRO=0.905, train_loss_epoch=-5.07e+5]
Epoch 2:  54%|█████▍    | 7/13 [00:09<00:08,  1.35s/it, loss=-4.99e+05, train_loss_step=-8.68e+5, pixel_AUROC=0.975, pixel_AUPRO=0.905, train_loss_epoch=-5.07e+5]
Epoch 2:  62%|██████▏   | 8/13 [00:09<00:05,  1.20s/it, loss=-4.99e+05, train_loss_step=-8.68e+5, pixel_AUROC=0.975, pixel_AUPRO=0.905, train_loss_epoch=-5.07e+5]
Epoch 2:  62%|██████▏   | 8/13 [00:09<00:06,  1.21s/it, loss=-5.53e+05, train_loss_step=-8.84e+5, pixel_AUROC=0.975, pixel_AUPRO=0.905, train_loss_epoch=-5.07e+5]
Epoch 2:  69%|██████▉   | 9/13 [00:10<00:04,  1.14s/it, loss=-5.53e+05, train_loss_step=-8.84e+5, pixel_AUROC=0.975, pixel_AUPRO=0.905, train_loss_epoch=-5.07e+5]
Epoch 2:  69%|██████▉   | 9/13 [00:10<00:04,  1.15s/it, loss=-6.02e+05, train_loss_step=-9.07e+5, pixel_AUROC=0.975, pixel_AUPRO=0.905, train_loss_epoch=-5.07e+5]

Validation: 0it [00:00, ?it/s]

Validation:   0%|          | 0/4 [00:00<?, ?it/s]

Validation DataLoader 0:   0%|          | 0/4 [00:00<?, ?it/s]

Validation DataLoader 0:  25%|██▌       | 1/4 [00:00<00:00,  6.70it/s]
Epoch 2:  77%|███████▋  | 10/13 [00:13<00:04,  1.35s/it, loss=-6.02e+05, train_loss_step=-9.07e+5, pixel_AUROC=0.975, pixel_AUPRO=0.905, train_loss_epoch=-5.07e+5]

Validation DataLoader 0:  50%|█████     | 2/4 [00:00<00:00,  6.06it/s]
Epoch 2:  85%|████████▍ | 11/13 [00:13<00:02,  1.24s/it, loss=-6.02e+05, train_loss_step=-9.07e+5, pixel_AUROC=0.975, pixel_AUPRO=0.905, train_loss_epoch=-5.07e+5]

Validation DataLoader 0:  75%|███████▌  | 3/4 [00:01<00:00,  1.67it/s]
Epoch 2:  92%|█████████▏| 12/13 [00:15<00:01,  1.26s/it, loss=-6.02e+05, train_loss_step=-9.07e+5, pixel_AUROC=0.975, pixel_AUPRO=0.905, train_loss_epoch=-5.07e+5]

Validation DataLoader 0: 100%|██████████| 4/4 [00:01<00:00,  2.13it/s]
Epoch 2: 100%|██████████| 13/13 [00:15<00:00,  1.17s/it, loss=-6.02e+05, train_loss_step=-9.07e+5, pixel_AUROC=0.975, pixel_AUPRO=0.905, train_loss_epoch=-5.07e+5]
Epoch 2: 100%|██████████| 13/13 [00:17<00:00,  1.33s/it, loss=-6.02e+05, train_loss_step=-9.07e+5, pixel_AUROC=0.983, pixel_AUPRO=0.933, train_loss_epoch=-5.07e+5]

                                                                      
Epoch 2: 100%|██████████| 13/13 [00:17<00:00,  1.33s/it, loss=-6.02e+05, train_loss_step=-9.07e+5, pixel_AUROC=0.983, pixel_AUPRO=0.933, train_loss_epoch=-7.94e+5]
Epoch 2:   0%|          | 0/13 [00:00<?, ?it/s, loss=-6.02e+05, train_loss_step=-9.07e+5, pixel_AUROC=0.983, pixel_AUPRO=0.933, train_loss_epoch=-7.94e+5]         
Epoch 3:   0%|          | 0/13 [00:00<?, ?it/s, loss=-6.02e+05, train_loss_step=-9.07e+5, pixel_AUROC=0.983, pixel_AUPRO=0.933, train_loss_epoch=-7.94e+5]
Epoch 3:   8%|▊         | 1/13 [00:03<00:38,  3.17s/it, loss=-6.02e+05, train_loss_step=-9.07e+5, pixel_AUROC=0.983, pixel_AUPRO=0.933, train_loss_epoch=-7.94e+5]
Epoch 3:   8%|▊         | 1/13 [00:03<00:39,  3.27s/it, loss=-6.47e+05, train_loss_step=-9.5e+5, pixel_AUROC=0.983, pixel_AUPRO=0.933, train_loss_epoch=-7.94e+5] 
Epoch 3:  15%|█▌        | 2/13 [00:03<00:18,  1.71s/it, loss=-6.47e+05, train_loss_step=-9.5e+5, pixel_AUROC=0.983, pixel_AUPRO=0.933, train_loss_epoch=-7.94e+5]
Epoch 3:  15%|█▌        | 2/13 [00:03<00:19,  1.77s/it, loss=-6.86e+05, train_loss_step=-9.86e+5, pixel_AUROC=0.983, pixel_AUPRO=0.933, train_loss_epoch=-7.94e+5]
Epoch 3:  23%|██▎       | 3/13 [00:05<00:17,  1.73s/it, loss=-6.86e+05, train_loss_step=-9.86e+5, pixel_AUROC=0.983, pixel_AUPRO=0.933, train_loss_epoch=-7.94e+5]
Epoch 3:  23%|██▎       | 3/13 [00:05<00:17,  1.77s/it, loss=-6.75e+05, train_loss_step=-4.5e+4, pixel_AUROC=0.983, pixel_AUPRO=0.933, train_loss_epoch=-7.94e+5] 
Epoch 3:  31%|███       | 4/13 [00:05<00:12,  1.36s/it, loss=-6.75e+05, train_loss_step=-4.5e+4, pixel_AUROC=0.983, pixel_AUPRO=0.933, train_loss_epoch=-7.94e+5]
Epoch 3:  31%|███       | 4/13 [00:05<00:12,  1.39s/it, loss=-6.95e+05, train_loss_step=-7.35e+5, pixel_AUROC=0.983, pixel_AUPRO=0.933, train_loss_epoch=-7.94e+5]
Epoch 3:  38%|███▊      | 5/13 [00:07<00:11,  1.45s/it, loss=-6.95e+05, train_loss_step=-7.35e+5, pixel_AUROC=0.983, pixel_AUPRO=0.933, train_loss_epoch=-7.94e+5]
Epoch 3:  38%|███▊      | 5/13 [00:07<00:11,  1.47s/it, loss=-6.92e+05, train_loss_step=-3.75e+5, pixel_AUROC=0.983, pixel_AUPRO=0.933, train_loss_epoch=-7.94e+5]
Epoch 3:  46%|████▌     | 6/13 [00:07<00:08,  1.25s/it, loss=-6.92e+05, train_loss_step=-3.75e+5, pixel_AUROC=0.983, pixel_AUPRO=0.933, train_loss_epoch=-7.94e+5]
Epoch 3:  46%|████▌     | 6/13 [00:07<00:08,  1.27s/it, loss=-7.05e+05, train_loss_step=-7.16e+5, pixel_AUROC=0.983, pixel_AUPRO=0.933, train_loss_epoch=-7.94e+5]
Epoch 3:  54%|█████▍    | 7/13 [00:09<00:07,  1.33s/it, loss=-7.05e+05, train_loss_step=-7.16e+5, pixel_AUROC=0.983, pixel_AUPRO=0.933, train_loss_epoch=-7.94e+5]
Epoch 3:  54%|█████▍    | 7/13 [00:09<00:08,  1.35s/it, loss=-7.19e+05, train_loss_step=-8.35e+5, pixel_AUROC=0.983, pixel_AUPRO=0.933, train_loss_epoch=-7.94e+5]
Epoch 3:  62%|██████▏   | 8/13 [00:09<00:05,  1.20s/it, loss=-7.19e+05, train_loss_step=-8.35e+5, pixel_AUROC=0.983, pixel_AUPRO=0.933, train_loss_epoch=-7.94e+5]
Epoch 3:  62%|██████▏   | 8/13 [00:09<00:06,  1.21s/it, loss=-7.33e+05, train_loss_step=-8.58e+5, pixel_AUROC=0.983, pixel_AUPRO=0.933, train_loss_epoch=-7.94e+5]
Epoch 3:  69%|██████▉   | 9/13 [00:10<00:04,  1.14s/it, loss=-7.33e+05, train_loss_step=-8.58e+5, pixel_AUROC=0.983, pixel_AUPRO=0.933, train_loss_epoch=-7.94e+5]
Epoch 3:  69%|██████▉   | 9/13 [00:10<00:04,  1.15s/it, loss=-7.45e+05, train_loss_step=-8.46e+5, pixel_AUROC=0.983, pixel_AUPRO=0.933, train_loss_epoch=-7.94e+5]

Validation: 0it [00:00, ?it/s]

Validation:   0%|          | 0/4 [00:00<?, ?it/s]

Validation DataLoader 0:   0%|          | 0/4 [00:00<?, ?it/s]

Validation DataLoader 0:  25%|██▌       | 1/4 [00:00<00:00,  6.51it/s]
Epoch 3:  77%|███████▋  | 10/13 [00:13<00:04,  1.37s/it, loss=-7.45e+05, train_loss_step=-8.46e+5, pixel_AUROC=0.983, pixel_AUPRO=0.933, train_loss_epoch=-7.94e+5]

Validation DataLoader 0:  50%|█████     | 2/4 [00:00<00:00,  5.90it/s]
Epoch 3:  85%|████████▍ | 11/13 [00:13<00:02,  1.26s/it, loss=-7.45e+05, train_loss_step=-8.46e+5, pixel_AUROC=0.983, pixel_AUPRO=0.933, train_loss_epoch=-7.94e+5]

Validation DataLoader 0:  75%|███████▌  | 3/4 [00:01<00:00,  1.66it/s]
Epoch 3:  92%|█████████▏| 12/13 [00:15<00:01,  1.28s/it, loss=-7.45e+05, train_loss_step=-8.46e+5, pixel_AUROC=0.983, pixel_AUPRO=0.933, train_loss_epoch=-7.94e+5]

Validation DataLoader 0: 100%|██████████| 4/4 [00:01<00:00,  2.12it/s]
Epoch 3: 100%|██████████| 13/13 [00:15<00:00,  1.19s/it, loss=-7.45e+05, train_loss_step=-8.46e+5, pixel_AUROC=0.983, pixel_AUPRO=0.933, train_loss_epoch=-7.94e+5]
Epoch 3: 100%|██████████| 13/13 [00:17<00:00,  1.35s/it, loss=-7.45e+05, train_loss_step=-8.46e+5, pixel_AUROC=0.967, pixel_AUPRO=0.891, train_loss_epoch=-7.94e+5]

                                                                      
Epoch 3: 100%|██████████| 13/13 [00:17<00:00,  1.35s/it, loss=-7.45e+05, train_loss_step=-8.46e+5, pixel_AUROC=0.967, pixel_AUPRO=0.891, train_loss_epoch=-7.01e+5]
Epoch 3:   0%|          | 0/13 [00:00<?, ?it/s, loss=-7.45e+05, train_loss_step=-8.46e+5, pixel_AUROC=0.967, pixel_AUPRO=0.891, train_loss_epoch=-7.01e+5]         
Epoch 4:   0%|          | 0/13 [00:00<?, ?it/s, loss=-7.45e+05, train_loss_step=-8.46e+5, pixel_AUROC=0.967, pixel_AUPRO=0.891, train_loss_epoch=-7.01e+5]
Epoch 4:   8%|▊         | 1/13 [00:03<00:41,  3.44s/it, loss=-7.45e+05, train_loss_step=-8.46e+5, pixel_AUROC=0.967, pixel_AUPRO=0.891, train_loss_epoch=-7.01e+5]
Epoch 4:   8%|▊         | 1/13 [00:03<00:42,  3.53s/it, loss=-7.54e+05, train_loss_step=-8.42e+5, pixel_AUROC=0.967, pixel_AUPRO=0.891, train_loss_epoch=-7.01e+5]
Epoch 4:  15%|█▌        | 2/13 [00:03<00:20,  1.85s/it, loss=-7.54e+05, train_loss_step=-8.42e+5, pixel_AUROC=0.967, pixel_AUPRO=0.891, train_loss_epoch=-7.01e+5]
Epoch 4:  15%|█▌        | 2/13 [00:03<00:20,  1.90s/it, loss=-7.59e+05, train_loss_step=-8.17e+5, pixel_AUROC=0.967, pixel_AUPRO=0.891, train_loss_epoch=-7.01e+5]
Epoch 4:  23%|██▎       | 3/13 [00:05<00:18,  1.82s/it, loss=-7.59e+05, train_loss_step=-8.17e+5, pixel_AUROC=0.967, pixel_AUPRO=0.891, train_loss_epoch=-7.01e+5]
Epoch 4:  23%|██▎       | 3/13 [00:05<00:18,  1.86s/it, loss=-7.65e+05, train_loss_step=-7.99e+5, pixel_AUROC=0.967, pixel_AUPRO=0.891, train_loss_epoch=-7.01e+5]
Epoch 4:  31%|███       | 4/13 [00:05<00:12,  1.43s/it, loss=-7.65e+05, train_loss_step=-7.99e+5, pixel_AUROC=0.967, pixel_AUPRO=0.891, train_loss_epoch=-7.01e+5]
Epoch 4:  31%|███       | 4/13 [00:05<00:13,  1.46s/it, loss=-7.71e+05, train_loss_step=-8.41e+5, pixel_AUROC=0.967, pixel_AUPRO=0.891, train_loss_epoch=-7.01e+5]
Epoch 4:  38%|███▊      | 5/13 [00:07<00:12,  1.50s/it, loss=-7.71e+05, train_loss_step=-8.41e+5, pixel_AUROC=0.967, pixel_AUPRO=0.891, train_loss_epoch=-7.01e+5]
Epoch 4:  38%|███▊      | 5/13 [00:07<00:12,  1.52s/it, loss=-7.79e+05, train_loss_step=-8.83e+5, pixel_AUROC=0.967, pixel_AUPRO=0.891, train_loss_epoch=-7.01e+5]
Epoch 4:  46%|████▌     | 6/13 [00:07<00:09,  1.30s/it, loss=-7.79e+05, train_loss_step=-8.83e+5, pixel_AUROC=0.967, pixel_AUPRO=0.891, train_loss_epoch=-7.01e+5]
Epoch 4:  46%|████▌     | 6/13 [00:07<00:09,  1.31s/it, loss=-7.87e+05, train_loss_step=-9.04e+5, pixel_AUROC=0.967, pixel_AUPRO=0.891, train_loss_epoch=-7.01e+5]
Epoch 4:  54%|█████▍    | 7/13 [00:09<00:08,  1.37s/it, loss=-7.87e+05, train_loss_step=-9.04e+5, pixel_AUROC=0.967, pixel_AUPRO=0.891, train_loss_epoch=-7.01e+5]
Epoch 4:  54%|█████▍    | 7/13 [00:09<00:08,  1.38s/it, loss=-7.91e+05, train_loss_step=-8.97e+5, pixel_AUROC=0.967, pixel_AUPRO=0.891, train_loss_epoch=-7.01e+5]
Epoch 4:  62%|██████▏   | 8/13 [00:09<00:06,  1.23s/it, loss=-7.91e+05, train_loss_step=-8.97e+5, pixel_AUROC=0.967, pixel_AUPRO=0.891, train_loss_epoch=-7.01e+5]
Epoch 4:  62%|██████▏   | 8/13 [00:09<00:06,  1.24s/it, loss=-6.63e+05, train_loss_step=1.73e+6, pixel_AUROC=0.967, pixel_AUPRO=0.891, train_loss_epoch=-7.01e+5] 
Epoch 4:  69%|██████▉   | 9/13 [00:10<00:04,  1.17s/it, loss=-6.63e+05, train_loss_step=1.73e+6, pixel_AUROC=0.967, pixel_AUPRO=0.891, train_loss_epoch=-7.01e+5]
Epoch 4:  69%|██████▉   | 9/13 [00:10<00:04,  1.17s/it, loss=-6.36e+05, train_loss_step=-3.2e+5, pixel_AUROC=0.967, pixel_AUPRO=0.891, train_loss_epoch=-7.01e+5]

Validation: 0it [00:00, ?it/s]

Validation:   0%|          | 0/4 [00:00<?, ?it/s]

Validation DataLoader 0:   0%|          | 0/4 [00:00<?, ?it/s]

Validation DataLoader 0:  25%|██▌       | 1/4 [00:00<00:00,  5.74it/s]
Epoch 4:  77%|███████▋  | 10/13 [00:14<00:04,  1.41s/it, loss=-6.36e+05, train_loss_step=-3.2e+5, pixel_AUROC=0.967, pixel_AUPRO=0.891, train_loss_epoch=-7.01e+5]

Validation DataLoader 0:  50%|█████     | 2/4 [00:00<00:00,  5.67it/s]
Epoch 4:  85%|████████▍ | 11/13 [00:14<00:02,  1.30s/it, loss=-6.36e+05, train_loss_step=-3.2e+5, pixel_AUROC=0.967, pixel_AUPRO=0.891, train_loss_epoch=-7.01e+5]

Validation DataLoader 0:  75%|███████▌  | 3/4 [00:01<00:00,  1.66it/s]
Epoch 4:  92%|█████████▏| 12/13 [00:15<00:01,  1.31s/it, loss=-6.36e+05, train_loss_step=-3.2e+5, pixel_AUROC=0.967, pixel_AUPRO=0.891, train_loss_epoch=-7.01e+5]

Validation DataLoader 0: 100%|██████████| 4/4 [00:01<00:00,  2.12it/s]
Epoch 4: 100%|██████████| 13/13 [00:15<00:00,  1.21s/it, loss=-6.36e+05, train_loss_step=-3.2e+5, pixel_AUROC=0.967, pixel_AUPRO=0.891, train_loss_epoch=-7.01e+5]
Epoch 4: 100%|██████████| 13/13 [00:17<00:00,  1.38s/it, loss=-6.36e+05, train_loss_step=-3.2e+5, pixel_AUROC=0.961, pixel_AUPRO=0.918, train_loss_epoch=-7.01e+5]

                                                                      
Epoch 4: 100%|██████████| 13/13 [00:17<00:00,  1.38s/it, loss=-6.36e+05, train_loss_step=-3.2e+5, pixel_AUROC=0.961, pixel_AUPRO=0.918, train_loss_epoch=-5.14e+5]
Epoch 4:   0%|          | 0/13 [00:00<?, ?it/s, loss=-6.36e+05, train_loss_step=-3.2e+5, pixel_AUROC=0.961, pixel_AUPRO=0.918, train_loss_epoch=-5.14e+5]         
Epoch 5:   0%|          | 0/13 [00:00<?, ?it/s, loss=-6.36e+05, train_loss_step=-3.2e+5, pixel_AUROC=0.961, pixel_AUPRO=0.918, train_loss_epoch=-5.14e+5]
Epoch 5:   8%|▊         | 1/13 [00:03<00:38,  3.22s/it, loss=-6.36e+05, train_loss_step=-3.2e+5, pixel_AUROC=0.961, pixel_AUPRO=0.918, train_loss_epoch=-5.14e+5]
Epoch 5:   8%|▊         | 1/13 [00:03<00:40,  3.34s/it, loss=-4.31e+05, train_loss_step=3.21e+6, pixel_AUROC=0.961, pixel_AUPRO=0.918, train_loss_epoch=-5.14e+5]
Epoch 5:  15%|█▌        | 2/13 [00:03<00:19,  1.75s/it, loss=-4.31e+05, train_loss_step=3.21e+6, pixel_AUROC=0.961, pixel_AUPRO=0.918, train_loss_epoch=-5.14e+5]
Epoch 5:  15%|█▌        | 2/13 [00:03<00:19,  1.80s/it, loss=-4.12e+05, train_loss_step=-5.16e+5, pixel_AUROC=0.961, pixel_AUPRO=0.918, train_loss_epoch=-5.14e+5]
Epoch 5:  23%|██▎       | 3/13 [00:05<00:17,  1.76s/it, loss=-4.12e+05, train_loss_step=-5.16e+5, pixel_AUROC=0.961, pixel_AUPRO=0.918, train_loss_epoch=-5.14e+5]
Epoch 5:  23%|██▎       | 3/13 [00:05<00:17,  1.79s/it, loss=-3.7e+05, train_loss_step=-1.15e+5, pixel_AUROC=0.961, pixel_AUPRO=0.918, train_loss_epoch=-5.14e+5] 
Epoch 5:  31%|███       | 4/13 [00:05<00:12,  1.39s/it, loss=-3.7e+05, train_loss_step=-1.15e+5, pixel_AUROC=0.961, pixel_AUPRO=0.918, train_loss_epoch=-5.14e+5]
Epoch 5:  31%|███       | 4/13 [00:05<00:12,  1.41s/it, loss=-3.62e+05, train_loss_step=-8.21e+5, pixel_AUROC=0.961, pixel_AUPRO=0.918, train_loss_epoch=-5.14e+5]
Epoch 5:  38%|███▊      | 5/13 [00:07<00:11,  1.46s/it, loss=-3.62e+05, train_loss_step=-8.21e+5, pixel_AUROC=0.961, pixel_AUPRO=0.918, train_loss_epoch=-5.14e+5]
Epoch 5:  38%|███▊      | 5/13 [00:07<00:11,  1.48s/it, loss=-4.02e+05, train_loss_step=-8.5e+5, pixel_AUROC=0.961, pixel_AUPRO=0.918, train_loss_epoch=-5.14e+5] 
Epoch 5:  46%|████▌     | 6/13 [00:07<00:08,  1.26s/it, loss=-4.02e+05, train_loss_step=-8.5e+5, pixel_AUROC=0.961, pixel_AUPRO=0.918, train_loss_epoch=-5.14e+5]
Epoch 5:  46%|████▌     | 6/13 [00:07<00:08,  1.28s/it, loss=-4.1e+05, train_loss_step=-8.93e+5, pixel_AUROC=0.961, pixel_AUPRO=0.918, train_loss_epoch=-5.14e+5]
Epoch 5:  54%|█████▍    | 7/13 [00:09<00:08,  1.35s/it, loss=-4.1e+05, train_loss_step=-8.93e+5, pixel_AUROC=0.961, pixel_AUPRO=0.918, train_loss_epoch=-5.14e+5]
Epoch 5:  54%|█████▍    | 7/13 [00:09<00:08,  1.36s/it, loss=-4.37e+05, train_loss_step=-9.19e+5, pixel_AUROC=0.961, pixel_AUPRO=0.918, train_loss_epoch=-5.14e+5]
Epoch 5:  62%|██████▏   | 8/13 [00:09<00:06,  1.20s/it, loss=-4.37e+05, train_loss_step=-9.19e+5, pixel_AUROC=0.961, pixel_AUPRO=0.918, train_loss_epoch=-5.14e+5]
Epoch 5:  62%|██████▏   | 8/13 [00:09<00:06,  1.22s/it, loss=-4.49e+05, train_loss_step=-9.49e+5, pixel_AUROC=0.961, pixel_AUPRO=0.918, train_loss_epoch=-5.14e+5]
Epoch 5:  69%|██████▉   | 9/13 [00:10<00:04,  1.15s/it, loss=-4.49e+05, train_loss_step=-9.49e+5, pixel_AUROC=0.961, pixel_AUPRO=0.918, train_loss_epoch=-5.14e+5]
Epoch 5:  69%|██████▉   | 9/13 [00:10<00:04,  1.15s/it, loss=-4.55e+05, train_loss_step=-9.71e+5, pixel_AUROC=0.961, pixel_AUPRO=0.918, train_loss_epoch=-5.14e+5]

Validation: 0it [00:00, ?it/s]

Validation:   0%|          | 0/4 [00:00<?, ?it/s]

Validation DataLoader 0:   0%|          | 0/4 [00:00<?, ?it/s]

Validation DataLoader 0:  25%|██▌       | 1/4 [00:00<00:00,  6.27it/s]
Epoch 5:  77%|███████▋  | 10/13 [00:13<00:04,  1.38s/it, loss=-4.55e+05, train_loss_step=-9.71e+5, pixel_AUROC=0.961, pixel_AUPRO=0.918, train_loss_epoch=-5.14e+5]

Validation DataLoader 0:  50%|█████     | 2/4 [00:00<00:00,  5.32it/s]
Epoch 5:  85%|████████▍ | 11/13 [00:14<00:02,  1.28s/it, loss=-4.55e+05, train_loss_step=-9.71e+5, pixel_AUROC=0.961, pixel_AUPRO=0.918, train_loss_epoch=-5.14e+5]

Validation DataLoader 0:  75%|███████▌  | 3/4 [00:01<00:00,  1.65it/s]
Epoch 5:  92%|█████████▏| 12/13 [00:15<00:01,  1.29s/it, loss=-4.55e+05, train_loss_step=-9.71e+5, pixel_AUROC=0.961, pixel_AUPRO=0.918, train_loss_epoch=-5.14e+5]

Validation DataLoader 0: 100%|██████████| 4/4 [00:01<00:00,  2.10it/s]
Epoch 5: 100%|██████████| 13/13 [00:15<00:00,  1.20s/it, loss=-4.55e+05, train_loss_step=-9.71e+5, pixel_AUROC=0.961, pixel_AUPRO=0.918, train_loss_epoch=-5.14e+5]
Epoch 5: 100%|██████████| 13/13 [00:17<00:00,  1.35s/it, loss=-4.55e+05, train_loss_step=-9.71e+5, pixel_AUROC=0.976, pixel_AUPRO=0.913, train_loss_epoch=-5.14e+5]

                                                                      
Epoch 5: 100%|██████████| 13/13 [00:17<00:00,  1.35s/it, loss=-4.55e+05, train_loss_step=-9.71e+5, pixel_AUROC=0.976, pixel_AUPRO=0.913, train_loss_epoch=-2.96e+5]
Epoch 5: 100%|██████████| 13/13 [00:17<00:00,  1.35s/it, loss=-4.55e+05, train_loss_step=-9.71e+5, pixel_AUROC=0.976, pixel_AUPRO=0.913, train_loss_epoch=-2.96e+5]

Testing: 0it [00:00, ?it/s]
Testing:   0%|          | 0/4 [00:00<?, ?it/s]
Testing DataLoader 0:   0%|          | 0/4 [00:00<?, ?it/s]
Testing DataLoader 0:  25%|██▌       | 1/4 [00:07<00:22,  7.61s/it]
Testing DataLoader 0:  50%|█████     | 2/4 [00:13<00:13,  6.78s/it]
Testing DataLoader 0:  75%|███████▌  | 3/4 [00:19<00:06,  6.51s/it]
Testing DataLoader 0: 100%|██████████| 4/4 [00:23<00:00,  5.89s/it]
Testing DataLoader 0: 100%|██████████| 4/4 [00:25<00:00,  6.33s/it]
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       Test metric             DataLoader 0
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       image_AUROC          0.9859550595283508
       pixel_AUPRO          1.2146450281143188
       pixel_AUROC           0.982062816619873
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Code of Conduct

blaz-r commented 10 months ago

Hello, thanks for reporting this. It's quite an unusual case. I'm not sure what the problem would be exactly, but may I just ask how you installed the anomalib? From your provided info:

Steps to reproduce the behavior Install anomalib pip install anomalib, in your python enviroment ... Anomalib version: 1.0.0.dev0

Did you install anomalib from github and then also with pip? This might be problematic, especially if you use v1, as the one from pip is I believe 0.7. Try doing a fresh install, but only from git OR from pip, not both.

Hope that helps 😄

samet-akcay commented 8 months ago

Any update here?

akshathamila commented 5 months ago

I am running into the same issue. I clone the repo and then: pip install -e . Then to install the remaining dependancies: anomalib install I am running this on a GPU.

blaz-r commented 5 months ago

This is a bit problematic as metrics should be reliable. Can you please provide more details (model used, data used) and give us a minimal reproducible code sample. Thanks 😄

akshathamila commented 5 months ago

Hi, Thank you for the quick response. I am using PatchCore with private data. I am not using the method from anomalib package directly, instead I am saving the normalized anomaly maps on the device and then calling the metric this way in a postprocessing script. I tried running on GPU and CPU. However, I get different metrics each time. aupro = AUPRO(fpr_limit=0.3) run_pro = aupro(preds_tensor, targets_tensor) run_pro tensor(1.0518)

blaz-r commented 5 months ago

Okay, thanks for the info. I'll look into this asap.

blaz-r commented 5 months ago

I can't reproduce this. I tried with Fastflow setup above but I get valid values. I think this is something very specific but the search area is so wide that I'm afraid I can't solve it without more exact information from your side. Is there any possibility you share the predictions and GT data?