search

churchlab / ml-ami-builder

Packer scripts to build nvidia-enabled AMIs
19 stars 4 forks source link

Continuously high CPU usage by systemd-udevd process #17

Closed glebkuznetsov closed 6 years ago

glebkuznetsov commented 6 years ago

Idle machines running at ~30% CPU usage.

Explicitly killing the process works (e.g. Identify pid using top or htop, then sudo kill <pid>.

Not sure what's causing this.

glebkuznetsov commented 6 years ago

Update: This only happens on CPU-only machines.

I think I've identified the problem being systemd attempting to run the nvidiaoptimize.sh script repeatedly and failing. I disabled it

sudo systemctl disable mlpe-nvidiaoptimize.service

Then restarted. Problem is gone.

Options to fix this:

  1. Separate AMI for CPU-only and GPU machines, leaving off the nvidia part.
  2. Update nvidiaoptimize.sh to run conditionally depending on being on CPU or GPU.

@grinner Opinion which one is easier?