search

mlco2 / codecarbon

Track emissions from Compute and recommend ways to reduce their impact on the environment.
https://mlco2.github.io/codecarbon
MIT License
1.18k stars 178 forks source link

Accuracy when deployed on Kubernetes #523

Closed accountForIssues closed 4 months ago

accountForIssues commented 8 months ago

Description

I have docker images that are used to train small ML models. These images are used to spin up containers on services such as Coreweave who use K8s for the same. I recently came to know about and added codecarbon to calculate the total resource usage. An example output (offline) is below:

{
  "timestamp": "2024-03-30T11:27:11",
  "project_name": "finetuning",
  "run_id": "84ad0f5d-5aab-4642-bd15-c6843671bf4c",
  "duration": 291.84384274482727,
  "emissions": 0.019239630322990336,
  "emissions_rate": 0.00006592440033011917,
  "cpu_power": 188.08739138414944,
  "gpu_power": 15.891167553168648,
  "ram_power": 283.29069471359253,
  "cpu_energy": 0.015584842142308182,
  "gpu_energy": 0.012249018965878378,
  "ram_energy": 0.022895533826661152,
  "energy_consumed": 0.050729394934847696,
  "country_name": "United States",
  "country_iso_code": "USA",
  "region": null,
  "cloud_provider": "",
  "cloud_region": "",
  "os": "Linux-5.19.17-coreweave-x86_64-with-glibc2.35",
  "python_version": "3.10.12",
  "codecarbon_version": "2.3.4",
  "cpu_count": 128,
  "cpu_model": "AMD EPYC 7513 32-Core Processor",
  "gpu_count": 1,
  "gpu_model": "1 x NVIDIA RTX A5000",
  "longitude": null,
  "latitude": null,
  "ram_total_size": 755.4418525695801,
  "tracking_mode": "machine",
  "on_cloud": "N",
  "pue": 1
}

The container is limited to 4x CPU, 16 GB RAM, 1x GPU. I am aware that CPU/RAM counts/sizes can be reflective of the entire system (esp. on Linux) so I'm not concerned about that.

What I am wondering is if the accuracy of the power usage of the different components and the emissions is affected because of the same ?

e.g., Looking up on Google, the CPU above has a TDP of 200W. So, for using 4 cores, 188W seems a lot. Similar with RAM, 283W for 16GB RAM. Though, GPU stats seems to be correct. And the GPU power usage seems to be incorrect. In the actual logs, it updated correctly with the actual usage but in the final emissions, it seems to wrong.

Am I incorrect in thinking that the final_emissions is the average or the overall stats ?

Is there anything I can do to "correct" this ? If not, can I use those numbers and calculate the right values ?

The complete log:

  warnings.warn(
[codecarbon INFO @ 11:22:18] offline tracker init
Starting training
/opt/runtime/lib/python3.10/site-packages/accelerate/accelerator.py:432: FutureWarning: Passing the following arguments to `Accelerator` is deprecated and will be removed in version 1.0 of Accelerate: dict_keys(['dispatch_batches', 'split_batches', 'even_batches', 'use_seedable_sampler']). Please pass an `accelerate.DataLoaderConfiguration` instead: 
dataloader_config = DataLoaderConfiguration(dispatch_batches=None, split_batches=False, even_batches=True, use_seedable_sampler=True)
  warnings.warn(
[codecarbon INFO @ 11:22:23] [setup] RAM Tracking...
[codecarbon INFO @ 11:22:23] [setup] GPU Tracking...
[codecarbon INFO @ 11:22:23] Tracking Nvidia GPU via pynvml
[codecarbon INFO @ 11:22:23] [setup] CPU Tracking...
[codecarbon INFO @ 11:22:23] Tracking Intel CPU via RAPL interface
[codecarbon INFO @ 11:22:24] >>> Tracker's metadata:
[codecarbon INFO @ 11:22:24]   Platform system: Linux-5.19.17-coreweave-x86_64-with-glibc2.35
[codecarbon INFO @ 11:22:24]   Python version: 3.10.12
[codecarbon INFO @ 11:22:24]   CodeCarbon version: 2.3.4
[codecarbon INFO @ 11:22:24]   Available RAM : 755.442 GB
[codecarbon INFO @ 11:22:24]   CPU count: 128
[codecarbon INFO @ 11:22:24]   CPU model: AMD EPYC 7513 32-Core Processor
[codecarbon INFO @ 11:22:24]   GPU count: 1
[codecarbon INFO @ 11:22:24]   GPU model: 1 x NVIDIA RTX A5000
[codecarbon INFO @ 11:22:35] Energy consumed for RAM : 0.001181 kWh. RAM Power : 283.29069471359253 W
[codecarbon INFO @ 11:22:35] Energy consumed for all GPUs : 0.000637 kWh. Total GPU Power : 152.92188567477933 W
[codecarbon INFO @ 11:22:35] Energy consumed for all CPUs : 0.000810 kWh. Total CPU Power : 194.3800774022706 W
[codecarbon INFO @ 11:22:35] 0.002628 kWh of electricity used since the beginning.
[codecarbon INFO @ 11:22:39] Energy consumed for RAM : 0.001181 kWh. RAM Power : 283.29069471359253 W
[codecarbon INFO @ 11:22:39] Energy consumed for all GPUs : 0.000881 kWh. Total GPU Power : 211.46567015674938 W
[codecarbon INFO @ 11:22:39] Energy consumed for all CPUs : 0.000818 kWh. Total CPU Power : 196.16573733791228 W
[codecarbon INFO @ 11:22:39] 0.002880 kWh of electricity used since the beginning.
[codecarbon INFO @ 11:22:50] Energy consumed for RAM : 0.002361 kWh. RAM Power : 283.29069471359253 W
[codecarbon INFO @ 11:22:50] Energy consumed for all GPUs : 0.001539 kWh. Total GPU Power : 216.40332782241614 W
[codecarbon INFO @ 11:22:50] Energy consumed for all CPUs : 0.001626 kWh. Total CPU Power : 195.8807353819956 W
[codecarbon INFO @ 11:22:50] 0.005526 kWh of electricity used since the beginning.
[codecarbon INFO @ 11:22:54] Energy consumed for RAM : 0.002361 kWh. RAM Power : 283.29069471359253 W
[codecarbon INFO @ 11:22:54] Energy consumed for all GPUs : 0.001785 kWh. Total GPU Power : 216.82075140786847 W
[codecarbon INFO @ 11:22:54] Energy consumed for all CPUs : 0.001626 kWh. Total CPU Power : 193.9645467722918 W
[codecarbon INFO @ 11:22:54] 0.005772 kWh of electricity used since the beginning.
[codecarbon INFO @ 11:23:05] Energy consumed for RAM : 0.003541 kWh. RAM Power : 283.29069471359253 W
[codecarbon INFO @ 11:23:05] Energy consumed for all GPUs : 0.002442 kWh. Total GPU Power : 216.69878876394674 W
[codecarbon INFO @ 11:23:05] Energy consumed for all CPUs : 0.002403 kWh. Total CPU Power : 186.50459183020192 W
[codecarbon INFO @ 11:23:05] 0.008386 kWh of electricity used since the beginning.
[codecarbon INFO @ 11:23:09] Energy consumed for RAM : 0.003541 kWh. RAM Power : 283.29069471359253 W
[codecarbon INFO @ 11:23:09] Energy consumed for all GPUs : 0.002687 kWh. Total GPU Power : 216.55376870699567 W
[codecarbon INFO @ 11:23:09] Energy consumed for all CPUs : 0.002408 kWh. Total CPU Power : 187.64311678359275 W
[codecarbon INFO @ 11:23:09] 0.008636 kWh of electricity used since the beginning.
[codecarbon INFO @ 11:23:20] Energy consumed for RAM : 0.004721 kWh. RAM Power : 283.29069471359253 W
[codecarbon INFO @ 11:23:20] Energy consumed for all GPUs : 0.003347 kWh. Total GPU Power : 217.33649972371734 W
[codecarbon INFO @ 11:23:20] Energy consumed for all CPUs : 0.003204 kWh. Total CPU Power : 192.23989274458245 W
[codecarbon INFO @ 11:23:20] 0.011273 kWh of electricity used since the beginning.
[codecarbon INFO @ 11:23:24] Energy consumed for RAM : 0.004721 kWh. RAM Power : 283.29069471359253 W
[codecarbon INFO @ 11:23:24] Energy consumed for all GPUs : 0.003595 kWh. Total GPU Power : 217.86490979107558 W
[codecarbon INFO @ 11:23:24] Energy consumed for all CPUs : 0.003219 kWh. Total CPU Power : 194.65922893781467 W
[codecarbon INFO @ 11:23:24] 0.011535 kWh of electricity used since the beginning.
[codecarbon INFO @ 11:23:35] Energy consumed for RAM : 0.005901 kWh. RAM Power : 283.29069471359253 W
[codecarbon INFO @ 11:23:35] Energy consumed for all GPUs : 0.004253 kWh. Total GPU Power : 217.34551139166553 W
[codecarbon INFO @ 11:23:35] Energy consumed for all CPUs : 0.004038 kWh. Total CPU Power : 200.16131429462683 W
[codecarbon INFO @ 11:23:35] 0.014193 kWh of electricity used since the beginning.
[codecarbon INFO @ 11:23:39] Energy consumed for RAM : 0.005902 kWh. RAM Power : 283.29069471359253 W
[codecarbon INFO @ 11:23:39] Energy consumed for all GPUs : 0.004505 kWh. Total GPU Power : 218.60975741052025 W
[codecarbon INFO @ 11:23:39] Energy consumed for all CPUs : 0.004036 kWh. Total CPU Power : 195.99686249519385 W
[codecarbon INFO @ 11:23:39] 0.014443 kWh of electricity used since the beginning.
[codecarbon INFO @ 11:23:50] Energy consumed for RAM : 0.007082 kWh. RAM Power : 283.29069471359253 W
[codecarbon INFO @ 11:23:50] Energy consumed for all GPUs : 0.005159 kWh. Total GPU Power : 217.53976149322924 W
[codecarbon INFO @ 11:23:50] Energy consumed for all CPUs : 0.004802 kWh. Total CPU Power : 183.3053337489947 W
[codecarbon INFO @ 11:23:50] 0.017043 kWh of electricity used since the beginning.
[codecarbon INFO @ 11:23:54] Energy consumed for RAM : 0.007082 kWh. RAM Power : 283.29069471359253 W
[codecarbon INFO @ 11:23:54] Energy consumed for all GPUs : 0.005412 kWh. Total GPU Power : 217.62620265723416 W
[codecarbon INFO @ 11:23:54] Energy consumed for all CPUs : 0.004817 kWh. Total CPU Power : 187.62387719600733 W
[codecarbon INFO @ 11:23:54] 0.017311 kWh of electricity used since the beginning.
[codecarbon INFO @ 11:24:05] Energy consumed for RAM : 0.008262 kWh. RAM Power : 283.29069471359253 W
[codecarbon INFO @ 11:24:05] Energy consumed for all GPUs : 0.006066 kWh. Total GPU Power : 217.67712470905812 W
[codecarbon INFO @ 11:24:05] Energy consumed for all CPUs : 0.005651 kWh. Total CPU Power : 203.80570501379754 W
[codecarbon INFO @ 11:24:05] 0.019979 kWh of electricity used since the beginning.
[codecarbon INFO @ 11:24:09] Energy consumed for RAM : 0.008262 kWh. RAM Power : 283.29069471359253 W
[codecarbon INFO @ 11:24:09] Energy consumed for all GPUs : 0.006319 kWh. Total GPU Power : 217.56044836397248 W
[codecarbon INFO @ 11:24:09] Energy consumed for all CPUs : 0.005678 kWh. Total CPU Power : 206.63374395997192 W
[codecarbon INFO @ 11:24:09] 0.020259 kWh of electricity used since the beginning.
[codecarbon INFO @ 11:24:20] Energy consumed for RAM : 0.009442 kWh. RAM Power : 283.29069471359253 W
[codecarbon INFO @ 11:24:20] Energy consumed for all GPUs : 0.006972 kWh. Total GPU Power : 217.50021930166227 W
[codecarbon INFO @ 11:24:20] Energy consumed for all CPUs : 0.006490 kWh. Total CPU Power : 201.18747378297917 W
[codecarbon INFO @ 11:24:20] 0.022904 kWh of electricity used since the beginning.
[codecarbon INFO @ 11:24:24] Energy consumed for RAM : 0.009442 kWh. RAM Power : 283.29069471359253 W
[codecarbon INFO @ 11:24:24] Energy consumed for all GPUs : 0.007226 kWh. Total GPU Power : 217.86609293215196 W
[codecarbon INFO @ 11:24:24] Energy consumed for all CPUs : 0.006510 kWh. Total CPU Power : 199.67553093575464 W
[codecarbon INFO @ 11:24:24] 0.023179 kWh of electricity used since the beginning.
[codecarbon INFO @ 11:24:35] Energy consumed for RAM : 0.010623 kWh. RAM Power : 283.29069471359253 W
[codecarbon INFO @ 11:24:35] Energy consumed for all GPUs : 0.007878 kWh. Total GPU Power : 217.41705050722248 W
[codecarbon INFO @ 11:24:35] Energy consumed for all CPUs : 0.007323 kWh. Total CPU Power : 200.03972500197082 W
[codecarbon INFO @ 11:24:35] 0.025824 kWh of electricity used since the beginning.
[codecarbon INFO @ 11:24:39] Energy consumed for RAM : 0.010623 kWh. RAM Power : 283.29069471359253 W
[codecarbon INFO @ 11:24:39] Energy consumed for all GPUs : 0.008131 kWh. Total GPU Power : 217.10246779263292 W
[codecarbon INFO @ 11:24:39] Energy consumed for all CPUs : 0.007321 kWh. Total CPU Power : 194.5743391067735 W
[codecarbon INFO @ 11:24:39] 0.026074 kWh of electricity used since the beginning.
[codecarbon INFO @ 11:24:50] Energy consumed for RAM : 0.011803 kWh. RAM Power : 283.29069471359253 W
[codecarbon INFO @ 11:24:50] Energy consumed for all GPUs : 0.008785 kWh. Total GPU Power : 217.63367077698433 W
[codecarbon INFO @ 11:24:50] Energy consumed for all CPUs : 0.008121 kWh. Total CPU Power : 191.3977514313449 W
[codecarbon INFO @ 11:24:50] 0.028708 kWh of electricity used since the beginning.
[codecarbon INFO @ 11:24:54] Energy consumed for RAM : 0.011803 kWh. RAM Power : 283.29069471359253 W
[codecarbon INFO @ 11:24:54] Energy consumed for all GPUs : 0.009037 kWh. Total GPU Power : 217.57754489427808 W
[codecarbon INFO @ 11:24:54] Energy consumed for all CPUs : 0.008117 kWh. Total CPU Power : 191.12425531593166 W
[codecarbon INFO @ 11:24:54] 0.028958 kWh of electricity used since the beginning.
[codecarbon INFO @ 11:25:05] Energy consumed for RAM : 0.012983 kWh. RAM Power : 283.29069471359253 W
[codecarbon INFO @ 11:25:05] Energy consumed for all GPUs : 0.009691 kWh. Total GPU Power : 217.5121731669453 W
[codecarbon INFO @ 11:25:05] Energy consumed for all CPUs : 0.008899 kWh. Total CPU Power : 186.80166610137837 W
[codecarbon INFO @ 11:25:05] 0.031573 kWh of electricity used since the beginning.
[codecarbon INFO @ 11:25:09] Energy consumed for RAM : 0.012983 kWh. RAM Power : 283.29069471359253 W
[codecarbon INFO @ 11:25:09] Energy consumed for all GPUs : 0.009944 kWh. Total GPU Power : 217.59776737455456 W
[codecarbon INFO @ 11:25:09] Energy consumed for all CPUs : 0.008897 kWh. Total CPU Power : 187.15978770314496 W
[codecarbon INFO @ 11:25:09] 0.031824 kWh of electricity used since the beginning.
[codecarbon INFO @ 11:25:20] Energy consumed for RAM : 0.014163 kWh. RAM Power : 283.29069471359253 W
[codecarbon INFO @ 11:25:20] Energy consumed for all GPUs : 0.010597 kWh. Total GPU Power : 217.38385352837207 W
[codecarbon INFO @ 11:25:20] Energy consumed for all CPUs : 0.009687 kWh. Total CPU Power : 189.04331044230418 W
[codecarbon INFO @ 11:25:20] 0.034447 kWh of electricity used since the beginning.
[codecarbon INFO @ 11:25:24] Energy consumed for RAM : 0.014163 kWh. RAM Power : 283.29069471359253 W
[codecarbon INFO @ 11:25:24] Energy consumed for all GPUs : 0.010850 kWh. Total GPU Power : 217.55496421541042 W
[codecarbon INFO @ 11:25:24] Energy consumed for all CPUs : 0.009676 kWh. Total CPU Power : 186.9530931490736 W
[codecarbon INFO @ 11:25:24] 0.034690 kWh of electricity used since the beginning.
[codecarbon INFO @ 11:25:35] Energy consumed for RAM : 0.015344 kWh. RAM Power : 283.29069471359253 W
[codecarbon INFO @ 11:25:35] Energy consumed for all GPUs : 0.011502 kWh. Total GPU Power : 217.3698502034015 W
[codecarbon INFO @ 11:25:35] Energy consumed for all CPUs : 0.010454 kWh. Total CPU Power : 184.15064997667594 W
[codecarbon INFO @ 11:25:35] 0.037300 kWh of electricity used since the beginning.
[codecarbon INFO @ 11:25:35] Energy consumed for RAM : 0.015033 kWh. RAM Power : 283.29069471359253 W
[codecarbon INFO @ 11:25:35] Energy consumed for all GPUs : 0.011513 kWh. Total GPU Power : 215.9679809175551 W
[codecarbon INFO @ 11:25:35] Energy consumed for all CPUs : 0.010244 kWh. Total CPU Power : 185.0518622461799 W
[codecarbon INFO @ 11:25:35] 0.036791 kWh of electricity used since the beginning.
/opt/runtime/lib/python3.10/site-packages/codecarbon/output.py:168: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat([df, pd.DataFrame.from_records([dict(data.values)])])
[codecarbon INFO @ 11:25:50] Energy consumed for RAM : 0.016531 kWh. RAM Power : 283.29069471359253 W
[codecarbon INFO @ 11:25:50] Energy consumed for all GPUs : 0.011830 kWh. Total GPU Power : 77.62553631713503 W
[codecarbon INFO @ 11:25:50] Energy consumed for all CPUs : 0.011266 kWh. Total CPU Power : 192.46517346170003 W
[codecarbon INFO @ 11:25:50] 0.039626 kWh of electricity used since the beginning.
[codecarbon INFO @ 11:26:05] Energy consumed for RAM : 0.017703 kWh. RAM Power : 283.29069471359253 W
[codecarbon INFO @ 11:26:05] Energy consumed for all GPUs : 0.011948 kWh. Total GPU Power : 28.522841009919272 W
[codecarbon INFO @ 11:26:05] Energy consumed for all CPUs : 0.012077 kWh. Total CPU Power : 195.97217970491823 W
[codecarbon INFO @ 11:26:05] 0.041728 kWh of electricity used since the beginning.
[codecarbon INFO @ 11:26:23] Energy consumed for RAM : 0.019145 kWh. RAM Power : 283.29069471359253 W
[codecarbon INFO @ 11:26:23] Energy consumed for all GPUs : 0.012036 kWh. Total GPU Power : 16.757284426890063 W
[codecarbon INFO @ 11:26:23] Energy consumed for all CPUs : 0.013043 kWh. Total CPU Power : 184.5241111439675 W
[codecarbon INFO @ 11:26:23] 0.044224 kWh of electricity used since the beginning.
[codecarbon INFO @ 11:26:36] Energy consumed for RAM : 0.020095 kWh. RAM Power : 283.29069471359253 W
[codecarbon INFO @ 11:26:36] Energy consumed for all GPUs : 0.012091 kWh. Total GPU Power : 16.46382340960936 W
[codecarbon INFO @ 11:26:36] Energy consumed for all CPUs : 0.013693 kWh. Total CPU Power : 192.9020442148626 W
[codecarbon INFO @ 11:26:36] 0.045879 kWh of electricity used since the beginning.
[codecarbon INFO @ 11:26:51] Energy consumed for RAM : 0.021276 kWh. RAM Power : 283.29069471359253 W
[codecarbon INFO @ 11:26:51] Energy consumed for all GPUs : 0.012159 kWh. Total GPU Power : 16.12618754811998 W
[codecarbon INFO @ 11:26:51] Energy consumed for all CPUs : 0.014516 kWh. Total CPU Power : 196.49352307184404 W
[codecarbon INFO @ 11:26:51] 0.047951 kWh of electricity used since the beginning.
[codecarbon INFO @ 11:27:06] Energy consumed for RAM : 0.022448 kWh. RAM Power : 283.29069471359253 W
[codecarbon INFO @ 11:27:06] Energy consumed for all GPUs : 0.012224 kWh. Total GPU Power : 15.794729809961883 W
[codecarbon INFO @ 11:27:06] Energy consumed for all CPUs : 0.015288 kWh. Total CPU Power : 186.65329813273098 W
[codecarbon INFO @ 11:27:06] 0.049960 kWh of electricity used since the beginning.
[codecarbon INFO @ 11:27:11] Energy consumed for RAM : 0.022896 kWh. RAM Power : 283.29069471359253 W
[codecarbon INFO @ 11:27:11] Energy consumed for all GPUs : 0.012249 kWh. Total GPU Power : 15.891167553168648 W
[codecarbon INFO @ 11:27:11] Energy consumed for all CPUs : 0.015585 kWh. Total CPU Power : 188.08739138414944 W
[codecarbon INFO @ 11:27:11] 0.050729 kWh of electricity used since the beginning.
/opt/runtime/lib/python3.10/site-packages/codecarbon/output.py:168: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat([df, pd.DataFrame.from_records([dict(data.values)])])
accountForIssues commented 8 months ago

Also tried it today with process tracking_mode. Besides the RAM, everything else seems to be similar to the above.

I'm mainly interested in the total emissions and energy used for the entire training (process) duration. The average power/energy will be nice to have but not necessary.

{
  "timestamp": "2024-03-31T13:05:42",
  "project_name": "finetuning",
  "run_id": "965fba31-6c53-4313-a8f4-7028837133de",
  "duration": 289.8846218585968,
  "emissions": 0.013126898727831899,
  "emissions_rate": 0.0000452831841981431,
  "cpu_power": 172.58601686518426,
  "gpu_power": 16.64991510011033,
  "ram_power": 1.6509547233581543,
  "cpu_energy": 0.019036346739134652,
  "gpu_energy": 0.015457212018537003,
  "ram_energy": 0.00011831364604057935,
  "energy_consumed": 0.034611872403712225,
  "country_name": "United States",
  "country_iso_code": "USA",
  "region": null,
  "cloud_provider": "",
  "cloud_region": "",
  "os": "Linux-5.19.17-coreweave-x86_64-with-glibc2.35",
  "python_version": "3.10.12",
  "codecarbon_version": "2.3.4",
  "cpu_count": 128,
  "cpu_model": "AMD EPYC 7513 32-Core Processor",
  "gpu_count": 1,
  "gpu_model": "1 x NVIDIA RTX A5000",
  "longitude": null,
  "latitude": null,
  "ram_total_size": 755.4418983459473,
  "tracking_mode": "process",
  "on_cloud": "N",
  "pue": 1.25
}

Complete log:

  warnings.warn(
[codecarbon INFO @ 13:00:51] offline tracker init
Starting training
/opt/runtime/lib/python3.10/site-packages/accelerate/accelerator.py:432: FutureWarning: Passing the following arguments to `Accelerator` is deprecated and will be removed in version 1.0 of Accelerate: dict_keys(['dispatch_batches', 'split_batches', 'even_batches', 'use_seedable_sampler']). Please pass an `accelerate.DataLoaderConfiguration` instead: 
dataloader_config = DataLoaderConfiguration(dispatch_batches=None, split_batches=False, even_batches=True, use_seedable_sampler=True)
  warnings.warn(
[codecarbon INFO @ 13:00:54] [setup] RAM Tracking...
[codecarbon INFO @ 13:00:54] [setup] GPU Tracking...
[codecarbon INFO @ 13:00:54] Tracking Nvidia GPU via pynvml
[codecarbon INFO @ 13:00:54] [setup] CPU Tracking...
[codecarbon INFO @ 13:00:54] Tracking Intel CPU via RAPL interface
[codecarbon INFO @ 13:00:55] >>> Tracker's metadata:
[codecarbon INFO @ 13:00:55]   Platform system: Linux-5.19.17-coreweave-x86_64-with-glibc2.35
[codecarbon INFO @ 13:00:55]   Python version: 3.10.12
[codecarbon INFO @ 13:00:55]   CodeCarbon version: 2.3.4
[codecarbon INFO @ 13:00:55]   Available RAM : 755.442 GB
[codecarbon INFO @ 13:00:55]   CPU count: 128
[codecarbon INFO @ 13:00:55]   CPU model: AMD EPYC 7513 32-Core Processor
[codecarbon INFO @ 13:00:55]   GPU count: 1
[codecarbon INFO @ 13:00:55]   GPU model: 1 x NVIDIA RTX A5000
[codecarbon INFO @ 13:01:07] Energy consumed for RAM : 0.000004 kWh. RAM Power : 0.7472505569458008 W
[codecarbon INFO @ 13:01:07] Energy consumed for all GPUs : 0.000844 kWh. Total GPU Power : 162.04537887489198 W
[codecarbon INFO @ 13:01:07] Energy consumed for all CPUs : 0.000976 kWh. Total CPU Power : 187.30510374925484 W
[codecarbon INFO @ 13:01:07] 0.001824 kWh of electricity used since the beginning.
[codecarbon INFO @ 13:01:11] Energy consumed for RAM : 0.001181 kWh. RAM Power : 283.2907118797302 W
[codecarbon INFO @ 13:01:11] Energy consumed for all GPUs : 0.000899 kWh. Total GPU Power : 215.70380513395017 W
[codecarbon INFO @ 13:01:11] Energy consumed for all CPUs : 0.000787 kWh. Total CPU Power : 188.89025180767266 W
[codecarbon INFO @ 13:01:11] 0.002867 kWh of electricity used since the beginning.
[codecarbon INFO @ 13:01:22] Energy consumed for RAM : 0.000008 kWh. RAM Power : 0.7472877502441406 W
[codecarbon INFO @ 13:01:22] Energy consumed for all GPUs : 0.001988 kWh. Total GPU Power : 219.68553636031692 W
[codecarbon INFO @ 13:01:22] Energy consumed for all CPUs : 0.001951 kWh. Total CPU Power : 187.1676668238832 W
[codecarbon INFO @ 13:01:22] 0.003947 kWh of electricity used since the beginning.
[codecarbon INFO @ 13:01:26] Energy consumed for RAM : 0.002361 kWh. RAM Power : 283.2907118797302 W
[codecarbon INFO @ 13:01:26] Energy consumed for all GPUs : 0.001816 kWh. Total GPU Power : 219.96875869661432 W
[codecarbon INFO @ 13:01:26] Energy consumed for all CPUs : 0.001556 kWh. Total CPU Power : 184.432795315955 W
[codecarbon INFO @ 13:01:26] 0.005733 kWh of electricity used since the beginning.
[codecarbon INFO @ 13:01:37] Energy consumed for RAM : 0.000012 kWh. RAM Power : 0.7472977638244629 W
[codecarbon INFO @ 13:01:37] Energy consumed for all GPUs : 0.003135 kWh. Total GPU Power : 220.09618502076347 W
[codecarbon INFO @ 13:01:37] Energy consumed for all CPUs : 0.002945 kWh. Total CPU Power : 190.95824251688984 W
[codecarbon INFO @ 13:01:37] 0.006091 kWh of electricity used since the beginning.
[codecarbon INFO @ 13:01:41] Energy consumed for RAM : 0.003541 kWh. RAM Power : 283.2907118797302 W
[codecarbon INFO @ 13:01:41] Energy consumed for all GPUs : 0.002732 kWh. Total GPU Power : 219.91277161568922 W
[codecarbon INFO @ 13:01:41] Energy consumed for all CPUs : 0.002351 kWh. Total CPU Power : 190.8530844741656 W
[codecarbon INFO @ 13:01:41] 0.008624 kWh of electricity used since the beginning.
[codecarbon INFO @ 13:01:52] Energy consumed for RAM : 0.000016 kWh. RAM Power : 0.7472991943359375 W
[codecarbon INFO @ 13:01:52] Energy consumed for all GPUs : 0.004280 kWh. Total GPU Power : 219.96788891210292 W
[codecarbon INFO @ 13:01:52] Energy consumed for all CPUs : 0.003912 kWh. Total CPU Power : 185.58129434690667 W
[codecarbon INFO @ 13:01:52] 0.008208 kWh of electricity used since the beginning.
[codecarbon INFO @ 13:01:56] Energy consumed for RAM : 0.004721 kWh. RAM Power : 283.2907118797302 W
[codecarbon INFO @ 13:01:56] Energy consumed for all GPUs : 0.003649 kWh. Total GPU Power : 220.19293419416147 W
[codecarbon INFO @ 13:01:56] Energy consumed for all CPUs : 0.003128 kWh. Total CPU Power : 186.34422525231454 W
[codecarbon INFO @ 13:01:56] 0.011498 kWh of electricity used since the beginning.
[codecarbon INFO @ 13:02:07] Energy consumed for RAM : 0.000019 kWh. RAM Power : 0.7472991943359375 W
[codecarbon INFO @ 13:02:07] Energy consumed for all GPUs : 0.005434 kWh. Total GPU Power : 221.63650961124793 W
[codecarbon INFO @ 13:02:07] Energy consumed for all CPUs : 0.004882 kWh. Total CPU Power : 186.37343104811384 W
[codecarbon INFO @ 13:02:07] 0.010336 kWh of electricity used since the beginning.
[codecarbon INFO @ 13:02:11] Energy consumed for RAM : 0.005902 kWh. RAM Power : 283.2907118797302 W
[codecarbon INFO @ 13:02:11] Energy consumed for all GPUs : 0.004566 kWh. Total GPU Power : 219.99364466007867 W
[codecarbon INFO @ 13:02:11] Energy consumed for all CPUs : 0.003926 kWh. Total CPU Power : 191.61053976239936 W
[codecarbon INFO @ 13:02:11] 0.014393 kWh of electricity used since the beginning.
[codecarbon INFO @ 13:02:22] Energy consumed for RAM : 0.000023 kWh. RAM Power : 0.7472991943359375 W
[codecarbon INFO @ 13:02:22] Energy consumed for all GPUs : 0.006580 kWh. Total GPU Power : 219.97876046345607 W
[codecarbon INFO @ 13:02:22] Energy consumed for all CPUs : 0.005918 kWh. Total CPU Power : 198.79787631332965 W
[codecarbon INFO @ 13:02:22] 0.012521 kWh of electricity used since the beginning.
[codecarbon INFO @ 13:02:26] Energy consumed for RAM : 0.007082 kWh. RAM Power : 283.2907118797302 W
[codecarbon INFO @ 13:02:26] Energy consumed for all GPUs : 0.005484 kWh. Total GPU Power : 220.35450828021658 W
[codecarbon INFO @ 13:02:26] Energy consumed for all CPUs : 0.004738 kWh. Total CPU Power : 194.79699729631926 W
[codecarbon INFO @ 13:02:26] 0.017304 kWh of electricity used since the beginning.
[codecarbon INFO @ 13:02:37] Energy consumed for RAM : 0.000027 kWh. RAM Power : 0.7473006248474121 W
[codecarbon INFO @ 13:02:37] Energy consumed for all GPUs : 0.007727 kWh. Total GPU Power : 220.30764672069915 W
[codecarbon INFO @ 13:02:37] Energy consumed for all CPUs : 0.006911 kWh. Total CPU Power : 190.61095360207042 W
[codecarbon INFO @ 13:02:37] 0.014665 kWh of electricity used since the beginning.
[codecarbon INFO @ 13:02:41] Energy consumed for RAM : 0.008262 kWh. RAM Power : 283.2907118797302 W
[codecarbon INFO @ 13:02:41] Energy consumed for all GPUs : 0.006401 kWh. Total GPU Power : 220.03266971569874 W
[codecarbon INFO @ 13:02:41] Energy consumed for all CPUs : 0.005536 kWh. Total CPU Power : 191.54620955104147 W
[codecarbon INFO @ 13:02:41] 0.020199 kWh of electricity used since the beginning.
[codecarbon INFO @ 13:02:52] Energy consumed for RAM : 0.000031 kWh. RAM Power : 0.7473034858703613 W
[codecarbon INFO @ 13:02:52] Energy consumed for all GPUs : 0.008876 kWh. Total GPU Power : 220.45023469092814 W
[codecarbon INFO @ 13:02:52] Energy consumed for all CPUs : 0.007907 kWh. Total CPU Power : 191.29399431133254 W
[codecarbon INFO @ 13:02:52] 0.016814 kWh of electricity used since the beginning.
[codecarbon INFO @ 13:02:56] Energy consumed for RAM : 0.009443 kWh. RAM Power : 283.2907118797302 W
[codecarbon INFO @ 13:02:56] Energy consumed for all GPUs : 0.007319 kWh. Total GPU Power : 220.47335336520013 W
[codecarbon INFO @ 13:02:56] Energy consumed for all CPUs : 0.006316 kWh. Total CPU Power : 187.31836655520834 W
[codecarbon INFO @ 13:02:56] 0.023078 kWh of electricity used since the beginning.
[codecarbon INFO @ 13:03:07] Energy consumed for RAM : 0.000035 kWh. RAM Power : 0.7473034858703613 W
[codecarbon INFO @ 13:03:07] Energy consumed for all GPUs : 0.010022 kWh. Total GPU Power : 220.19263817540437 W
[codecarbon INFO @ 13:03:07] Energy consumed for all CPUs : 0.008880 kWh. Total CPU Power : 186.72056421455744 W
[codecarbon INFO @ 13:03:07] 0.018937 kWh of electricity used since the beginning.
[codecarbon INFO @ 13:03:11] Energy consumed for RAM : 0.010623 kWh. RAM Power : 283.2907118797302 W
[codecarbon INFO @ 13:03:11] Energy consumed for all GPUs : 0.008235 kWh. Total GPU Power : 219.70956299804305 W
[codecarbon INFO @ 13:03:11] Energy consumed for all CPUs : 0.007103 kWh. Total CPU Power : 188.6756998997984 W
[codecarbon INFO @ 13:03:11] 0.025960 kWh of electricity used since the beginning.
[codecarbon INFO @ 13:03:22] Energy consumed for RAM : 0.000039 kWh. RAM Power : 0.7473034858703613 W
[codecarbon INFO @ 13:03:22] Energy consumed for all GPUs : 0.011168 kWh. Total GPU Power : 219.91025199628967 W
[codecarbon INFO @ 13:03:22] Energy consumed for all CPUs : 0.009831 kWh. Total CPU Power : 182.67233940762765 W
[codecarbon INFO @ 13:03:22] 0.021037 kWh of electricity used since the beginning.
[codecarbon INFO @ 13:03:26] Energy consumed for RAM : 0.011802 kWh. RAM Power : 283.2907118797302 W
[codecarbon INFO @ 13:03:26] Energy consumed for all GPUs : 0.009153 kWh. Total GPU Power : 220.69038322081815 W
[codecarbon INFO @ 13:03:26] Energy consumed for all CPUs : 0.007875 kWh. Total CPU Power : 185.39338430812245 W
[codecarbon INFO @ 13:03:26] 0.028830 kWh of electricity used since the beginning.
[codecarbon INFO @ 13:03:37] Energy consumed for RAM : 0.000043 kWh. RAM Power : 0.74857234954834 W
[codecarbon INFO @ 13:03:37] Energy consumed for all GPUs : 0.012317 kWh. Total GPU Power : 220.62472577864406 W
[codecarbon INFO @ 13:03:37] Energy consumed for all CPUs : 0.010828 kWh. Total CPU Power : 191.53017824020802 W
[codecarbon INFO @ 13:03:37] 0.023188 kWh of electricity used since the beginning.
[codecarbon INFO @ 13:03:41] Energy consumed for RAM : 0.012982 kWh. RAM Power : 283.2907118797302 W
[codecarbon INFO @ 13:03:41] Energy consumed for all GPUs : 0.010071 kWh. Total GPU Power : 220.35166275717611 W
[codecarbon INFO @ 13:03:41] Energy consumed for all CPUs : 0.008664 kWh. Total CPU Power : 189.4216319957635 W
[codecarbon INFO @ 13:03:41] 0.031718 kWh of electricity used since the beginning.
[codecarbon INFO @ 13:03:52] Energy consumed for RAM : 0.000047 kWh. RAM Power : 0.7485051155090332 W
[codecarbon INFO @ 13:03:52] Energy consumed for all GPUs : 0.013464 kWh. Total GPU Power : 220.3341024353972 W
[codecarbon INFO @ 13:03:52] Energy consumed for all CPUs : 0.011807 kWh. Total CPU Power : 187.8630951608062 W
[codecarbon INFO @ 13:03:52] 0.025318 kWh of electricity used since the beginning.
[codecarbon INFO @ 13:03:56] Energy consumed for RAM : 0.014163 kWh. RAM Power : 283.2907118797302 W
[codecarbon INFO @ 13:03:56] Energy consumed for all GPUs : 0.010991 kWh. Total GPU Power : 220.68522910907973 W
[codecarbon INFO @ 13:03:56] Energy consumed for all CPUs : 0.009432 kWh. Total CPU Power : 184.23930973773656 W
[codecarbon INFO @ 13:03:56] 0.034585 kWh of electricity used since the beginning.
[codecarbon INFO @ 13:04:05] Energy consumed for RAM : 0.014877 kWh. RAM Power : 283.2907118797302 W
[codecarbon INFO @ 13:04:05] Energy consumed for all GPUs : 0.011548 kWh. Total GPU Power : 220.89852844422032 W
[codecarbon INFO @ 13:04:05] Energy consumed for all CPUs : 0.009920 kWh. Total CPU Power : 193.70954231999332 W
[codecarbon INFO @ 13:04:05] 0.036345 kWh of electricity used since the beginning.
/opt/runtime/lib/python3.10/site-packages/codecarbon/output.py:168: FutureWarning: The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.
  df = pd.concat([df, pd.DataFrame.from_records([dict(data.values)])])
[codecarbon INFO @ 13:04:07] Energy consumed for RAM : 0.000051 kWh. RAM Power : 0.7462048530578613 W
[codecarbon INFO @ 13:04:07] Energy consumed for all GPUs : 0.014553 kWh. Total GPU Power : 209.16505605331326 W
[codecarbon INFO @ 13:04:07] Energy consumed for all CPUs : 0.012783 kWh. Total CPU Power : 187.30186791343655 W
[codecarbon INFO @ 13:04:07] 0.027387 kWh of electricity used since the beginning.
[codecarbon INFO @ 13:04:22] Energy consumed for RAM : 0.000057 kWh. RAM Power : 1.2548003196716309 W
[codecarbon INFO @ 13:04:22] Energy consumed for all GPUs : 0.014896 kWh. Total GPU Power : 65.63594918902683 W
[codecarbon INFO @ 13:04:22] Energy consumed for all CPUs : 0.013801 kWh. Total CPU Power : 195.25563044500802 W
[codecarbon INFO @ 13:04:22] 0.028753 kWh of electricity used since the beginning.
[codecarbon INFO @ 13:04:37] Energy consumed for RAM : 0.000064 kWh. RAM Power : 1.2580332756042483 W
[codecarbon INFO @ 13:04:37] Energy consumed for all GPUs : 0.015019 kWh. Total GPU Power : 23.65649601202967 W
[codecarbon INFO @ 13:04:37] Energy consumed for all CPUs : 0.014796 kWh. Total CPU Power : 189.7860203271394 W
[codecarbon INFO @ 13:04:37] 0.029878 kWh of electricity used since the beginning.
[codecarbon INFO @ 13:04:54] Energy consumed for RAM : 0.000073 kWh. RAM Power : 1.5729517936706543 W
[codecarbon INFO @ 13:04:54] Energy consumed for all GPUs : 0.015161 kWh. Total GPU Power : 23.752753694724493 W
[codecarbon INFO @ 13:04:54] Energy consumed for all CPUs : 0.015913 kWh. Total CPU Power : 186.81443076718745 W
[codecarbon INFO @ 13:04:54] 0.031147 kWh of electricity used since the beginning.
[codecarbon INFO @ 13:05:09] Energy consumed for RAM : 0.000087 kWh. RAM Power : 2.7783265113830566 W
[codecarbon INFO @ 13:05:09] Energy consumed for all GPUs : 0.015257 kWh. Total GPU Power : 18.331170484957525 W
[codecarbon INFO @ 13:05:09] Energy consumed for all CPUs : 0.016889 kWh. Total CPU Power : 185.26499250280602 W
[codecarbon INFO @ 13:05:09] 0.032234 kWh of electricity used since the beginning.
[codecarbon INFO @ 13:05:25] Energy consumed for RAM : 0.000108 kWh. RAM Power : 3.921996116638184 W
[codecarbon INFO @ 13:05:25] Energy consumed for all GPUs : 0.015356 kWh. Total GPU Power : 17.928908913098713 W
[codecarbon INFO @ 13:05:25] Energy consumed for all CPUs : 0.017953 kWh. Total CPU Power : 193.35531902638104 W
[codecarbon INFO @ 13:05:25] 0.033417 kWh of electricity used since the beginning.
[codecarbon INFO @ 13:05:41] Energy consumed for RAM : 0.000118 kWh. RAM Power : 1.9352631568908691 W
[codecarbon INFO @ 13:05:41] Energy consumed for all GPUs : 0.015451 kWh. Total GPU Power : 17.816804237108826 W
[codecarbon INFO @ 13:05:41] Energy consumed for all CPUs : 0.018970 kWh. Total CPU Power : 191.06551666798623 W
[codecarbon INFO @ 13:05:41] 0.034538 kWh of electricity used since the beginning.
[codecarbon INFO @ 13:05:42] Energy consumed for RAM : 0.000118 kWh. RAM Power : 1.6509547233581543 W
[codecarbon INFO @ 13:05:42] Energy consumed for all GPUs : 0.015457 kWh. Total GPU Power : 16.64991510011033 W
[codecarbon INFO @ 13:05:42] Energy consumed for all CPUs : 0.019036 kWh. Total CPU Power : 172.58601686518426 W
[codecarbon INFO @ 13:05:42] 0.034612 kWh of electricity used since the beginning.
LuisBlanche commented 8 months ago

Hello, can you provide a little more details about why you think the results are incorrect ?

Am I incorrect in thinking that the final_emissions is the average or the overall stats ?

Yes that seems incorrect, the final emissions are computed from the sum of the consumption that you get for every interval if you measure every 1 min for 10 minutes and you get around 0.02kWh each minutes then the average is 0.02kWh, but the total is 10*0.02 = 0.2kWh

benoit-cty commented 8 months ago

As you can see in the log Available RAM : 755.442 GB so we use this size for the estimation.

Same for the CPU, it is a 32 code AMD EPYC 7513 32-Core Processor so we monitor all of them.

What could help you is to use the code from https://github.com/mlco2/codecarbon/pull/316 than allows to monitor only the process that launch CodeCarbon.

benoit-cty commented 4 months ago

I'm closing this PR, don't hesitate to re-open if you need more information.