Closed yjy5921591ok closed 3 years ago
The error log is printed as follows:
2021/04/25 14:09:15 ERROR: Cpuset for the containers of Pod: cpupod with ID: f9c0efcc-cd48-4431-bf85-bb0959aa5e26 could not be re-adjusted, because: cpuset file does not exist for container: ace2008b27c43b31933ab995a3677a005417bf3efcd27788d935405dd1af9838 under the provided cgroupfs hierarchy: /sys/fs/cgroup/cpuset/kubepods
2021/04/25 14:09:15 error unmarshalling kubelet checkpoint file: json: cannot unmarshal object into Go struct field checkpointPodDevicesEntry.Data.PodDeviceEntries.DeviceIDs of type []string
2021/04/25 14:09:15 ERROR: Cpuset for the containers of Pod: cpupod ID: f9c0efcc-cd48-4431-bf85-bb0959aa5e26 could not be re-adjusted, because: json: cannot unmarshal object into Go struct field checkpointPodDevicesEntry.Data.PodDeviceEntries.DeviceIDs of type []string
2021/04/25 14:09:15 ERROR: Cpuset for the containers of Pod: cpupod with ID: f9c0efcc-cd48-4431-bf85-bb0959aa5e26 could not be re-adjusted, because: cpuset file does not exist for container: 5c60ffa4a3b8f86c27bd9dab6972ea35c236adf05261388abaf82dc1e7ff2b01 under the provided cgroupfs hierarchy: /sys/fs/cgroup/cpuset/kubepods
@yjy5921591ok : can you try what happens if you ask for exclusive cores from exclusive-numa-0 pool instead?
based on the "error unmarshalling kubelet checkpoint file: json: cannot unmarshal object into Go struct field checkpointPodDevicesEntry.Data.PodDeviceEntries.DeviceIDs of type []string" error I'm thinking the problem might be with resource pools which have only one entry. it can be K8s doesn't consistently marshal this information into JSON when the DeviceID map has only one entry
The current use of numa-0 also has similar problems.I have two errors here, the unmarshall error, and the other cgroup error. Are these two errors due to the same reason?In addition to the master branch, 0.3.0 also has this problem. My current version of k8s is 1.21. How can I avoid this problem? Do I still need to adjust the k8s configuration, or what other measures need to be implemented?First of all I want to make cpu-pooler run normally,may you guys can give me a guide.Thanks
yes the two can be related, but might not be. note, we are currently refactoring the CPU setting part (https://github.com/nokia/CPU-Pooler/pull/62) because in some cases it can happen the events from a Pod coming up do not reach us in time, so by the time we start changing its cpuset the container already disappeared with the latest master this should be only an intermittent error, and even when it happens it should self-correct itself. usually latest after 2-3 automated restarts the container comes up on its own
if your failure is permanent, it prob has a different root cause, and that might be the unmarshalling error. I have never seen this one personally, but also we currently use 1.20, not 1.21. the syntax might have changed? can you share (or upload somewhere) the "/var/lib/kubelet/device-plugins/kubelet_internal_checkpoint" file from the node you observe the failure?
Thank you very much for your prompt response, this error has been resolved. It is because my local cgroup path is different from the path mounted in cpusetter. This path is automatically generated when k8s is deployed, and may be a version difference. Anyway, thank you for your answers. I will continue to pay attention to this project.
Describe the bug Cpu affinity conflicts with cgroups configuration when create pod with exclusive-pool
To Reproduce Steps to reproduce the behavior: 1.Download the compressed package of the master branch code and copy it to the k8s environment 2.Compile and install the components of cpu-pooler according to official instructions 3.For the convenience of installation, I put the generated bin file, mirror image and dependent yaml file in the same folder, and wrote a shell script. The script content is as follows:
4.The custom configmap is as follows:
5.So far, all pods related to cpu-pooler seem to be running normally:
The allocation of resources seems to be successful:
6.Then I created a pod based on the test file given in the source package and adjusted it for my own configmap:
7.This pod contains 3 containers, the exclusivetest container will always try to restart, and the other two containers are normal. The error of the exclusivetest container is as follows:
8.Resources should be allocated correctly:
9.But the cgroups configuration seems to be incorrect: vi /sys/fs/cgroup/cpuset/kubepods/cpuset.cpus(The result of this command is null) vi /sys/fs/cgroup/cpuset/cpuset.cpus(The result of this command is 0-7)
Additional context This project greatly facilitates cpu binding operations, and I want to use this project to improve my platform. Your guidance is very important to me. Thank you for your contribution.