search

mlco2 / codecarbon

Track emissions from Compute and recommend ways to reduce their impact on the environment.
https://mlco2.github.io/codecarbon
MIT License
1.01k stars 158 forks source link

RAM power consumption #439

Open AKHCE opened 11 months ago

AKHCE commented 11 months ago

Hi

As far as I learned from documentation, codecarbone estimates ram power consumption based on a rule of thumb: "RAM CodeCarbon uses a 3 Watts for 8 BG ratio source . This mesure is not satisfying and if ever you have an idea how to enhance it please do not hesitate to contribute." So, codecarbone doesn't measure power consumption of ram directly.

If one install Intel® Power Gadget, there is panel showing specifically ram power. Given the fact that codecarbon uses Intel® Power Gadget in background, my question is why codecarbon relies on the rule of thumb above for ram power consumption and why codecarbon doesn't use Intel® Power Gadget for ramp power consumption.

Thank you so much

benoit-cty commented 11 months ago

Hello, Thanks for pointing this out. It seems that it depends on the CPU (or chipset ?) you are using, but it will be a great improvement. The reason we did not do it is just a lack of contributor to the project. If you want to contribute on this I will be happy to help you.

For the record, here is the code we use for CPU with Intel® Power Gadget : https://github.com/mlco2/codecarbon/blob/9a840204eda830735e8817e708a7b4d90d0f7bff/codecarbon/external/hardware.py#L172

AKHCE commented 11 months ago

Thank you for your reply. You're referring to a good point that does the ram power consumption depend on cpu or not. I've been investigating .....

benoit-cty commented 8 months ago

Another example of RAM estimation problem: I'm running my code on a cluster and CodeCarbon see the total RAM of the cluster, and all the CPU:

[codecarbon INFO @ 08:18:04]   CodeCarbon version: 2.3.1
[codecarbon INFO @ 08:18:04]   Available RAM : 1007.352 GB
[codecarbon INFO @ 08:18:04]   CPU count: 104
[codecarbon INFO @ 08:18:04]   CPU model: Intel(R) Xeon(R) Gold 5320 CPU @ 2.20GHz
[codecarbon INFO @ 08:18:04]   GPU count: 1
[codecarbon INFO @ 08:18:04]   GPU model: 1 x NVIDIA A2

So the estimations give 377W for RAM and 59W for GPU while training a model:

------------------------
[codecarbon INFO @ 08:24:24] Energy consumed for RAM : 0.039332 kWh. RAM Power : 377.75717210769653 W
[codecarbon DEBUG @ 08:24:24] RAM : 377.76 W during 14.99 s [measurement time: 0.0005]
[codecarbon INFO @ 08:24:24] Energy consumed for all GPUs : 0.006199 kWh. Total GPU Power : 59.644690577283775 W
[codecarbon DEBUG @ 08:24:24] GPU : 59.64 W during 14.99 s [measurement time: 0.0027]
[codecarbon INFO @ 08:24:24] Energy consumed for all CPUs : 0.009636 kWh. Total CPU Power : 92.5 W
[codecarbon DEBUG @ 08:24:24] CPU : 92.50 W during 15.00 s [measurement time: 0.0001]
[codecarbon INFO @ 08:24:24] 0.055167 kWh of electricity used since the beginning.
[codecarbon DEBUG @ 08:24:24] last_duration=14.994339942932129
------------------------

When adding tracking_mode = process it's better: [codecarbon INFO @ 12:07:20] Energy consumed for RAM : 0.000051 kWh. RAM Power : 1.5230913162231445 W

benoit-cty commented 8 months ago

A way to have info on ram units : sudo lshw -C memory -short | grep DIMM Sample output:

/0/37/0                                    memory         8GiB DIMM DDR4 Synchrone Unbuffered (Unregistered) 2400 MHz (0,4 ns)
/0/37/1                                    memory         8GiB DIMM DDR4 Synchrone Unbuffered (Unregistered) 2400 MHz (0,4 ns)
/0/37/2                                    memory         8GiB DIMM DDR4 Synchrone Unbuffered (Unregistered) 2400 MHz (0,4 ns)
/0/37/3                                    memory         8GiB DIMM DDR4 Synchrone Unbuffered (Unregistered) 2400 MHz (0,4 ns)