fix cpuset cgroup manager initialization

While tracking down another bug, I got really confused by log messages that you'll only see on the first run of a Nomad agent on a new system, caused by some spaghetti code in how we bootstrap the cpuset manager on the client.

(the error content is from a branch)

    2022-08-22T21:30:21.284-0500 [WARN]  client: failed to lookup cpuset from cgroup parent, and not set as reservable_cores: parent=nomad.slice error="openat2 /sys/fs/cgroup/nomad.slice/cpuset.cpus.effective: no such file or directory"
    2022-08-22T21:30:21.284-0500 [DEBUG] client.cpuset.v2: initializing with: cores=[]
    2022-08-22T21:30:21.300-0500 [DEBUG] client.cpuset.v2: establish cgroup hierarchy: parent=nomad.slice

This block will attempt to retrieve the effective cpuset (if reservable cores are not set) - before actually ensuring the parent cgroup for nomad exists. This means on first run, we get the warning message above and an empty set of usable cores.

// Ensure cgroups are created on linux platform
if runtime.GOOS == "linux" && c.cpusetManager != nil {
    // use the client configuration for reservable_cores if set
    cores := conf.ReservableCores
    if len(cores) == 0 {
        // otherwise lookup the effective cores from the parent cgroup
        cores, err = cgutil.GetCPUsFromCgroup(conf.CgroupParent)
        if err != nil {
            c.logger.Warn("failed to lookup cpuset from cgroup parent, and not set as reservable_cores", "parent", conf.CgroupParent)
            // will continue with a disabled cpuset manager
        }
    }
    if cpuErr := c.cpusetManager.Init(cores); cpuErr != nil {
        // If the client cannot initialize the cgroup then reserved cores will not be reported and the cpuset manager
        // will be disabled. this is common when running in dev mode under a non-root user for example.
        c.logger.Warn("failed to initialize cpuset cgroup subsystem, cpuset management disabled", "error", cpuErr)
        c.cpusetManager = new(cgutil.NoopCpusetManager)
    }
}

The empty set of useable cores means tasks configured to run will not have a value set in their cpuset.cpus cgroup.

cgroup/nomad.slice/594f8fba-c4e1-0e6a-38c3-b056db697bfa.py1.scope 
➜ cat cpuset.cpus

cgroup/nomad.slice/594f8fba-c4e1-0e6a-38c3-b056db697bfa.py1.scope

Any restart of the Nomad agent afterwords and the bug is gone - the parent cgroup will exist and the initialization will work as intended.

hashicorp / nomad

fix cpuset cgroup manager initialization #14229