search

lfwa / carbontracker

Track and predict the energy consumption and carbon footprint of training deep learning models.
MIT License
352 stars 26 forks source link

Issue monitoring i7 CPU #40

Closed raghavian closed 1 year ago

raghavian commented 3 years ago

Trying to use carbontracker on a new workstation on Ubuntu 20.04 (Intel i7-10700F CPU) and throws the following error:

with open(os.path.join(path, "energy_uj"), 'r') as f:
PermissionError: [Errno 13] Permission denied: '/sys/class/powercap/intel-rapl:0/energy_uj'

Most likely due to RAPL. Works when components="gpu" only.

raghavian commented 3 years ago

Looks like this is a Linux kernel update issue. To use carbontracker without this error, give read permission to the specific file:

sudo chmod 444 /sys/class/powercap/intel-rapl:0/energy_uj

If you do not have root access, ignore CPU monitoring when instantiating carbontracker:

tracker = CarbonTracker(epochs=args.num_epochs,components="gpu",
            log_dir='carbontracker/',monitor_epochs=-1)
hrdl-github commented 1 year ago

To make this change persistent, consider adding a udev rule, e.g.

# /etc/udev/rules.d/powercap.rules
ACTION=="add|change", SUBSYSTEM=="powercap", KERNEL=="intel-rapl:*", RUN+="/bin/chmod og+r %S%p/energy_uj"
hrdl-github commented 1 year ago

To immediately apply the permission changes:

sudo udevadm control --reload && sudo udevadm trigger --subsystem-match=powercap
raghavian commented 1 year ago

This will be an issue for users when they don't have root privileges. Will reopen the issue until we figure out a different solution.

PedramBakh commented 1 year ago

Release 1.1.7 addresses this issue by informing the user of missing privileges and refering to this issue thread.