huggingface / optimum-benchmark

🏋️ A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of Optimum's hardware optimizations & quantization schemes.
Apache License 2.0
255 stars 48 forks source link

Error: Another instance of codecarbon is already running #260

Closed j-irion closed 2 months ago

j-irion commented 2 months ago

Hello, When I try to track the energy consumption with the benchmark, I get the following error: Error: Another instance of codecarbon is already running. Turn off the other instance to be able to run this one. Exiting.

Afterwards I get another Error: AttributeError: 'EmissionsTracker' object has no attribute '_scheduler'

I noticed that, when I reboot the machine, the first run crashes later than the consequent runs and also saves to the task_codecarbon.json. Here are the logs of the first and second run: first_run_after_reboot.log second_run.log

I tried running the benchmark with the following config:

defaults:
  - benchmark
  - scenario: inference
  - launcher: process
  - backend: pytorch
  - _base_
  - _self_

name: pytorch_llama_8b_energy

launcher:
  device_isolation: true
  device_isolation_action: warn

backend:
  model: meta-llama/Meta-Llama-3.1-8B
  device: cuda
  device_ids: 0,1
  device_map: auto
  torch_dtype: float16

scenario:
  memory: true
  latency: true
  energy: true

  warmup_runs: 10
  iterations: 10
  duration: 10

  input_shapes:
    batch_size: 2
    sequence_length: 512
  generate_kwargs:
    max_new_tokens: 32
    min_new_tokens: 32
IlyasMoutawwakil commented 2 months ago

Hi, it must be the new release of codecarbon that's breaking stuff https://pypi.org/project/codecarbon/#history, I'll see if it can be fixed as part of the #169

j-irion commented 2 months ago

Thanks for your effort! Do you know, by any chance, which was the last compatible version of codecarbon?

IlyasMoutawwakil commented 2 months ago

2.6 probably since, our tests were passing then.