[Task]: Comet error during hyperparamater optimization

[Renat2001]

What is the motivation for this task?

I would like to find appropriate hyper parameters using Comet. I run python tools/hpo/sweep.py --model padim --model_config custom_padim.yaml --sweep_config tools/hpo/configs/comet.yaml. custom_padim.yaml:

  name: tumar
  format: folder
  path: ../data
  normal_dir: good # name of the folder containing normal images.
  abnormal_dir: bad # name of the folder containing abnormal images.
  normal_test_dir: null # name of the folder containing normal test images.
  task: classification # classification or segmentation
  mask: null #optional
  extensions: null
  split_ratio: 0.2 # ratio of the normal images that will be used to create a test split
  image_size: 256
  train_batch_size: 32
  test_batch_size: 32
  num_workers: 8
  normalization: imagenet # data distribution to which the images will be normalized: [none, imagenet]
  test_split_mode: from_dir # options: [from_dir, synthetic]
  val_split_mode: same_as_test # options: [same_as_test, from_test, sythetic]
  val_split_ratio: 0.5 # fraction of train/test images held out for validation (usage depends on val_split_mode)
  transform_config:
    train: null
    val: null
  create_validation_set: true
  tiling:
    apply: false
    tile_size: null
    stride: null
    remove_border_count: 0
    use_random_tiling: False
    random_tile_count: 16

model:
  name: padim
  backbone: resnet18
  pre_trained: true
  layers:
    - layer1
    - layer2
    - layer3
  normalization_method: min_max # options: [none, min_max, cdf]

metrics:
  image:
    - F1Score
    - AUROC
  pixel:
    - F1Score
    - AUROC
  threshold:
    method: adaptive #options: [adaptive, manual]
    manual_image: null
    manual_pixel: null

visualization:
  show_images: False # show images on the screen
  save_images: False # save images to the file system
  log_images: True # log images to the available loggers (if any)
  image_save_path: null # path to which images will be saved
  mode: full # options: ["full", "simple"]

project:
  seed: 42
  path: ./results

logging:
  logger: [comet] # options: [comet, tensorboard, wandb, csv] or combinations.
  log_graph: True # Logs the model graph to respective logger.

optimization:
  export_mode: null # options: torch, onnx, openvino

# PL Trainer Args. Don't add extra parameter here.
trainer:
  enable_checkpointing: true
  default_root_dir: null
  gradient_clip_val: 0
  gradient_clip_algorithm: norm
  num_nodes: 1
  devices: [1]
  enable_progress_bar: true
  overfit_batches: 0.0
  track_grad_norm: -1
  check_val_every_n_epoch: 1 # Don't validate before extracting features.
  fast_dev_run: false
  accumulate_grad_batches: 1
  max_epochs: 1
  min_epochs: null
  max_steps: -1
  min_steps: null
  max_time: null
  limit_train_batches: 1.0
  limit_val_batches: 1.0
  limit_test_batches: 1.0
  limit_predict_batches: 1.0
  val_check_interval: 1.0 # Don't validate before extracting features.
  log_every_n_steps: 50
  accelerator: gpu # <"cpu", "gpu", "tpu", "ipu", "hpu", "auto">
  strategy: null
  sync_batchnorm: false
  precision: 32
  enable_model_summary: true
  num_sanity_val_steps: 0
  profiler: null
  benchmark: false
  deterministic: false
  reload_dataloaders_every_n_epochs: 0
  auto_lr_find: false
  replace_sampler_ddp: true
  detect_anomaly: false
  auto_scale_batch_size: false
  plugins: null
  move_metrics_to_cpu: false
  multiple_trainloader_mode: max_size_cycle

comet.iaml:

algorithm: "bayes"
spec:
  maxCombo: 10
  metric: "image_F1Score"
  objective: "maximize"
parameters:
  dataset:
    category: capsule
    image_size:
      type: discrete
      values: [128, 256]
  model:
    backbone:
      type: categorical
      values: ["resnet18", "wide_resnet50_2"]

What I get:

OpenVINO is not installed. Please install OpenVINO to use OpenVINOInferencer.
/home/r.abdrakhmanov/tumar/padim/anomalib/src/anomalib/config/config.py:280: UserWarning: config.project.unique_dir is set to False. This does not ensure that your results will be written in an empty directory and you may overwrite files.
  warn(
Global seed set to 42
COMET INFO: 3f29a474fd714c8caf01957a0a9a47f4
COMET INFO: Using optimizer config: {'algorithm': 'bayes', 'configSpaceSize': 4, 'endTime': None, 'id': '3f29a474fd714c8caf01957a0a9a47f4', 'lastUpdateTime': None, 'maxCombo': 10, 'name': '3f29a474fd714c8caf01957a0a9a47f4', 'parameters': {'dataset.image_size': {'type': 'discrete', 'values': [128, 256]}, 'model.backbone': {'type': 'categorical', 'values': ['resnet18', 'wide_resnet50_2']}}, 'predictor': None, 'spec': {'gridSize': 10, 'maxCombo': 10, 'metric': 'image_F1Score', 'minSampleSize': 100, 'objective': 'maximize', 'retryAssignLimit': 0, 'retryLimit': 1000}, 'startTime': 23994265, 'state': {'mode': None, 'seed': None, 'sequence': [], 'sequence_i': 0, 'sequence_pid': None, 'sequence_retry': 0, 'sequence_retry_count': 0}, 'status': 'running', 'suggestion_count': 0, 'trials': 1, 'version': '2.0.23'}
COMET ERROR: Optimizer failed to retrieve data from the optimizer service, for more information look at: https://comet.com/docs/v2/api-and-sdk/python-sdk/warnings-errors/

Describe the solution you'd like

I would like to find appropriate hyper parameters using Comet

Additional context

No response

[xuzhengzhenghit]

I have also encountered the same problem. Have the bloggers resolved it?