search

tkestack / gpu-manager

Other
830 stars 235 forks source link

[need help] Failed to create a deployment requesting 0.x GPU cores #144

Open guofengtd opened 2 years ago

guofengtd commented 2 years ago

hello guys,

when I create a deployment with one GPU card, everything goes well

but, if with GPU that less than 1, it fails

bellow are the logs and node info, please kindly review them, and supply any suggestion or solution, thanks.

deployment log

2022-01-27T13:41:21.696682567+08:00 F0127 05:41:21.696330 7 client.go:78] fail to get response from manager, error rpc error: code = Unknown desc = can't find kubepods-burstable-podf96b0b34_8379_4581_9c87_2e98637f2fc2.slice from docker
2022-01-27T13:41:21.698605243+08:00 /tmp/cuda-control/src/register.c:87 rpc client exit with 255

node info

% kubectl describe node
Name:               192.168.41.104
Roles:              master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=192.168.41.104
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/master=
                    nvidia-device-enable=enable
                    platform.tkestack.io/machine-ip=192.168.41.104
Annotations:        flannel.alpha.coreos.com/backend-data: {"VtepMAC":"62:f1:6f:67:5d:d5"}
                    flannel.alpha.coreos.com/backend-type: vxlan
                    flannel.alpha.coreos.com/kube-subnet-manager: true
                    flannel.alpha.coreos.com/public-ip: 192.168.41.104
                    kubeadm.alpha.kubernetes.io/cri-socket: /var/run/containerd/containerd.sock
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Tue, 25 Jan 2022 10:23:41 +0800
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  192.168.41.104
  AcquireTime:     <unset>
  RenewTime:       Thu, 27 Jan 2022 15:06:29 +0800
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Thu, 27 Jan 2022 15:04:50 +0800   Tue, 25 Jan 2022 10:23:37 +0800   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Thu, 27 Jan 2022 15:04:50 +0800   Tue, 25 Jan 2022 10:23:37 +0800   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Thu, 27 Jan 2022 15:04:50 +0800   Tue, 25 Jan 2022 10:23:37 +0800   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Thu, 27 Jan 2022 15:04:50 +0800   Wed, 26 Jan 2022 18:16:45 +0800   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  192.168.41.104
  Hostname:    192.168.41.104
Capacity:
  cpu:                       48
  ephemeral-storage:         307050000Ki
  hugepages-1Gi:             0
  hugepages-2Mi:             0
  memory:                    32660268Ki
  pods:                      256
  tencent.com/vcuda-core:    100
  tencent.com/vcuda-memory:  62
Allocatable:
  cpu:                       47800m
  ephemeral-storage:         282977279532
  hugepages-1Gi:             0
  hugepages-2Mi:             0
  memory:                    31533868Ki
  pods:                      256
  tencent.com/vcuda-core:    100
  tencent.com/vcuda-memory:  62
System Info:
  Machine ID:                 9ffa73bff8a9485ea512c442e554d291
  System UUID:                00000000-0000-0000-0000-AC1F6B450EAE
  Boot ID:                    28318fa8-fe7d-4b63-a1eb-3665b240af1a
  Kernel Version:             3.10.0-1160.53.1.el7.x86_64
  OS Image:                   CentOS Linux 7 (Core)
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.5.4
  Kubelet Version:            v1.21.4-tke.1
  Kube-Proxy Version:         v1.21.4-tke.1
PodCIDR:                      172.16.0.0/24
PodCIDRs:                     172.16.0.0/24
Non-terminated Pods:          (47 in total)
  Namespace                   Name                                           CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                           ------------  ----------  ---------------  -------------  ---
  default                     empty9-8547ff7fd-2dmkf                         250m (0%)     500m (1%)   256Mi (0%)       1Gi (3%)       86m
  kube-system                 coredns-6c654f5cc6-58bz2                       100m (0%)     0 (0%)      70Mi (0%)        170Mi (0%)     2d4h
  kube-system                 coredns-6c654f5cc6-wsg7s                       100m (0%)     0 (0%)      70Mi (0%)        170Mi (0%)     2d4h
  kube-system                 etcd-192.168.41.104                            100m (0%)     0 (0%)      100Mi (0%)       0 (0%)         2d4h
  kube-system                 flannel-pmfcv                                  100m (0%)     100m (0%)   50Mi (0%)        256Mi (0%)     2d4h
  kube-system                 galaxy-daemonset-tc6t6                         100m (0%)     0 (0%)      200Mi (0%)       0 (0%)         2d4h
  kube-system                 gpu-manager-daemonset-89827                    0 (0%)        0 (0%)      0 (0%)           0 (0%)         4h46m
  kube-system                 gpu-quota-admission-76cfff49b6-gs8b5           1 (2%)        2 (4%)      1Gi (3%)         2Gi (6%)       2d4h
  kube-system                 kube-apiserver-192.168.41.104                  250m (0%)     0 (0%)      0 (0%)           0 (0%)         2d4h
  kube-system                 kube-controller-manager-192.168.41.104         200m (0%)     0 (0%)      0 (0%)           0 (0%)         2d4h
  kube-system                 kube-proxy-26g7w                               0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d4h
  kube-system                 kube-scheduler-192.168.41.104                  100m (0%)     0 (0%)      0 (0%)           0 (0%)         2d4h
  kube-system                 metrics-server-v0.3.6-794ccd69c8-8sqxb         87m (0%)      182m (0%)   154Mi (0%)       404Mi (1%)     2d4h
  tke                         influxdb-0                                     0 (0%)        0 (0%)      0 (0%)           0 (0%)         2d4h
  tke                         tke-application-api-6db6c5f8dd-l98gc           50m (0%)      500m (1%)   128Mi (0%)       1Gi (3%)       2d4h
  tke                         tke-application-api-6db6c5f8dd-tbf2x           50m (0%)      500m (1%)   128Mi (0%)       1Gi (3%)       2d4h
  tke                         tke-application-controller-6c769cc9cc-8v4zc    50m (0%)      100m (0%)   128Mi (0%)       256Mi (0%)     2d4h
  tke                         tke-application-controller-6c769cc9cc-lbf2s    50m (0%)      100m (0%)   128Mi (0%)       256Mi (0%)     2d4h
  tke                         tke-auth-api-6fcfdb454d-kv7bb                  250m (0%)     500m (1%)   256Mi (0%)       512Mi (1%)     2d4h
  tke                         tke-auth-api-6fcfdb454d-srxtq                  250m (0%)     500m (1%)   256Mi (0%)       512Mi (1%)     2d4h
  tke                         tke-auth-controller-5fbbdd9dd9-npjrd           50m (0%)      100m (0%)   128Mi (0%)       256Mi (0%)     2d4h
  tke                         tke-auth-controller-5fbbdd9dd9-qj5fx           50m (0%)      100m (0%)   128Mi (0%)       256Mi (0%)     2d4h
  tke                         tke-business-api-77ff89ff7-kwcnw               50m (0%)      100m (0%)   128Mi (0%)       256Mi (0%)     2d4h
  tke                         tke-business-api-77ff89ff7-s94ml               50m (0%)      100m (0%)   128Mi (0%)       256Mi (0%)     2d4h
  tke                         tke-business-controller-86879f65df-5wf94       50m (0%)      100m (0%)   128Mi (0%)       256Mi (0%)     2d4h
  tke                         tke-business-controller-86879f65df-rz2kv       50m (0%)      100m (0%)   128Mi (0%)       256Mi (0%)     2d4h
  tke                         tke-gateway-4gcpl                              50m (0%)      100m (0%)   128Mi (0%)       256Mi (0%)     2d4h
  tke                         tke-logagent-api-6d9b6fc4-7mg2r                50m (0%)      100m (0%)   128Mi (0%)       256Mi (0%)     2d4h
  tke                         tke-logagent-api-6d9b6fc4-fbsqk                50m (0%)      100m (0%)   128Mi (0%)       256Mi (0%)     2d4h
  tke                         tke-logagent-controller-c6c8bcbf6-gc6z6        50m (0%)      100m (0%)   128Mi (0%)       256Mi (0%)     2d4h
  tke                         tke-logagent-controller-c6c8bcbf6-wc7bf        50m (0%)      100m (0%)   128Mi (0%)       256Mi (0%)     2d4h
  tke                         tke-monitor-api-54bc7f44f6-7pl4s               50m (0%)      100m (0%)   128Mi (0%)       256Mi (0%)     2d4h
  tke                         tke-monitor-api-54bc7f44f6-xxcl5               50m (0%)      100m (0%)   128Mi (0%)       256Mi (0%)     2d4h
  tke                         tke-monitor-controller-bb56765d-5vl2x          50m (0%)      100m (0%)   128Mi (0%)       256Mi (0%)     2d4h
  tke                         tke-monitor-controller-bb56765d-78p6v          50m (0%)      100m (0%)   128Mi (0%)       256Mi (0%)     2d4h
  tke                         tke-notify-api-df68cb499-hzc8w                 50m (0%)      100m (0%)   128Mi (0%)       256Mi (0%)     2d4h
  tke                         tke-notify-api-df68cb499-qtnwz                 50m (0%)      100m (0%)   128Mi (0%)       256Mi (0%)     2d4h
  tke                         tke-notify-controller-85cf89b5bb-ngkqf         50m (0%)      100m (0%)   128Mi (0%)       256Mi (0%)     2d4h
  tke                         tke-notify-controller-85cf89b5bb-p9bbx         50m (0%)      100m (0%)   128Mi (0%)       256Mi (0%)     2d4h
  tke                         tke-platform-api-b4d45f4bc-4z5ln               100m (0%)     150m (0%)   256Mi (0%)       512Mi (1%)     2d4h
  tke                         tke-platform-api-b4d45f4bc-j5dbq               100m (0%)     150m (0%)   256Mi (0%)       512Mi (1%)     2d4h
  tke                         tke-platform-controller-546fc4fc97-phzgj       50m (0%)      100m (0%)   128Mi (0%)       1Gi (3%)       2d4h
  tke                         tke-platform-controller-546fc4fc97-sbp9l       50m (0%)      100m (0%)   128Mi (0%)       1Gi (3%)       2d4h
  tke                         tke-registry-api-55b6f778f7-j849x              50m (0%)      500m (1%)   128Mi (0%)       1Gi (3%)       2d4h
  tke                         tke-registry-api-55b6f778f7-mkhwn              50m (0%)      500m (1%)   128Mi (0%)       1Gi (3%)       2d4h
  tke                         tke-registry-controller-5bbb898bc5-hk6rw       50m (0%)      100m (0%)   128Mi (0%)       256Mi (0%)     2d4h
  tke                         tke-registry-controller-5bbb898bc5-r8p26       50m (0%)      100m (0%)   128Mi (0%)       256Mi (0%)     2d4h
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                  Requests      Limits
  --------                  --------      ------
  cpu                       4537m (9%)    8582m (17%)
  memory                    6660Mi (21%)  18152Mi (58%)
  ephemeral-storage         0 (0%)        0 (0%)
  hugepages-1Gi             0 (0%)        0 (0%)
  hugepages-2Mi             0 (0%)        0 (0%)
  tencent.com/vcuda-core    20            20
  tencent.com/vcuda-memory  3             3
Events:                     <none>

GPU-Manager log

2022-01-27T15:14:57.338651164+08:00 copy /usr/local/host/lib/libnvidia-ml.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:14:57.353867616+08:00 copy /usr/local/host/lib/libnvidia-ml.so.1 to /usr/local/nvidia/lib
2022-01-27T15:14:57.365707593+08:00 copy /usr/local/host/lib/libnvidia-ml.so to /usr/local/nvidia/lib
2022-01-27T15:14:57.377306514+08:00 copy /usr/local/host/lib64/libnvidia-ml.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:14:57.390168998+08:00 copy /usr/local/host/lib64/libnvidia-ml.so.1 to /usr/local/nvidia/lib64
2022-01-27T15:14:57.403748079+08:00 copy /usr/local/host/lib64/libnvidia-ml.so to /usr/local/nvidia/lib64
2022-01-27T15:14:57.752285035+08:00 copy /usr/local/host/lib/libcuda.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:14:57.789709907+08:00 copy /usr/local/host/lib/libcuda.so.1 to /usr/local/nvidia/lib
2022-01-27T15:14:57.801813172+08:00 copy /usr/local/host/lib/libcuda.so to /usr/local/nvidia/lib
2022-01-27T15:14:57.814628166+08:00 copy /usr/local/host/lib64/libcuda.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:14:57.855645410+08:00 copy /usr/local/host/lib64/libcuda.so.1 to /usr/local/nvidia/lib64
2022-01-27T15:14:57.868238721+08:00 copy /usr/local/host/lib64/libcuda.so to /usr/local/nvidia/lib64
2022-01-27T15:14:58.221402375+08:00 copy /usr/local/host/lib/libnvidia-ptxjitcompiler.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:14:58.246832945+08:00 copy /usr/local/host/lib/libnvidia-ptxjitcompiler.so.1 to /usr/local/nvidia/lib
2022-01-27T15:14:58.260925497+08:00 copy /usr/local/host/lib/libnvidia-ptxjitcompiler.so to /usr/local/nvidia/lib
2022-01-27T15:14:58.272734588+08:00 copy /usr/local/host/lib64/libnvidia-ptxjitcompiler.so to /usr/local/nvidia/lib64
2022-01-27T15:14:58.284625993+08:00 copy /usr/local/host/lib64/libnvidia-ptxjitcompiler.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:14:58.306610183+08:00 copy /usr/local/host/lib64/libnvidia-ptxjitcompiler.so.1 to /usr/local/nvidia/lib64
2022-01-27T15:14:59.021397422+08:00 copy /usr/local/host/lib/libnvidia-opencl.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:14:59.053452117+08:00 copy /usr/local/host/lib/libnvidia-opencl.so.1 to /usr/local/nvidia/lib
2022-01-27T15:14:59.064877049+08:00 copy /usr/local/host/lib64/libnvidia-opencl.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:14:59.093746142+08:00 copy /usr/local/host/lib64/libnvidia-opencl.so.1 to /usr/local/nvidia/lib64
2022-01-27T15:14:59.436163522+08:00 copy /usr/local/host/lib/libnvidia-compiler.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:14:59.506278506+08:00 copy /usr/local/host/lib64/libnvidia-compiler.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:14:59.887614862+08:00 copy /usr/local/host/lib/vdpau/libvdpau_nvidia.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:14:59.900631544+08:00 copy /usr/local/host/lib/vdpau/libvdpau_nvidia.so.1 to /usr/local/nvidia/lib
2022-01-27T15:14:59.914070936+08:00 copy /usr/local/host/lib/libvdpau_nvidia.so to /usr/local/nvidia/lib
2022-01-27T15:14:59.925889044+08:00 copy /usr/local/host/lib64/vdpau/libvdpau_nvidia.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:14:59.937774449+08:00 copy /usr/local/host/lib64/vdpau/libvdpau_nvidia.so.1 to /usr/local/nvidia/lib64
2022-01-27T15:14:59.949215912+08:00 copy /usr/local/host/lib64/libvdpau_nvidia.so to /usr/local/nvidia/lib64
2022-01-27T15:15:00.276571393+08:00 copy /usr/local/host/lib/libnvidia-encode.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:15:00.289498988+08:00 copy /usr/local/host/lib/libnvidia-encode.so.1 to /usr/local/nvidia/lib
2022-01-27T15:15:00.302716100+08:00 copy /usr/local/host/lib/libnvidia-encode.so to /usr/local/nvidia/lib
2022-01-27T15:15:00.314625777+08:00 copy /usr/local/host/lib64/libnvidia-encode.so to /usr/local/nvidia/lib64
2022-01-27T15:15:00.326298946+08:00 copy /usr/local/host/lib64/libnvidia-encode.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:15:00.337656884+08:00 copy /usr/local/host/lib64/libnvidia-encode.so.1 to /usr/local/nvidia/lib64
2022-01-27T15:15:00.630485163+08:00 copy /usr/local/host/lib/libnvcuvid.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:15:00.646156009+08:00 copy /usr/local/host/lib/libnvcuvid.so.1 to /usr/local/nvidia/lib
2022-01-27T15:15:00.658620776+08:00 copy /usr/local/host/lib/libnvcuvid.so to /usr/local/nvidia/lib
2022-01-27T15:15:00.669256945+08:00 copy /usr/local/host/lib64/libnvcuvid.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:15:00.685615089+08:00 copy /usr/local/host/lib64/libnvcuvid.so.1 to /usr/local/nvidia/lib64
2022-01-27T15:15:00.696418920+08:00 copy /usr/local/host/lib64/libnvcuvid.so to /usr/local/nvidia/lib64
2022-01-27T15:15:00.986894243+08:00 copy /usr/local/host/lib/libnvidia-fbc.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:15:00.998607169+08:00 copy /usr/local/host/lib/libnvidia-fbc.so.1 to /usr/local/nvidia/lib
2022-01-27T15:15:01.010264318+08:00 copy /usr/local/host/lib/libnvidia-fbc.so to /usr/local/nvidia/lib
2022-01-27T15:15:01.022598489+08:00 copy /usr/local/host/lib64/libnvidia-fbc.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:15:01.033919311+08:00 copy /usr/local/host/lib64/libnvidia-fbc.so.1 to /usr/local/nvidia/lib64
2022-01-27T15:15:01.044758481+08:00 copy /usr/local/host/lib64/libnvidia-fbc.so to /usr/local/nvidia/lib64
2022-01-27T15:15:01.397140614+08:00 copy /usr/local/host/lib/libnvidia-ifr.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:15:01.410247941+08:00 copy /usr/local/host/lib/libnvidia-ifr.so.1 to /usr/local/nvidia/lib
2022-01-27T15:15:01.421835077+08:00 copy /usr/local/host/lib/libnvidia-ifr.so to /usr/local/nvidia/lib
2022-01-27T15:15:01.435419169+08:00 copy /usr/local/host/lib64/libnvidia-ifr.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:15:01.447542578+08:00 copy /usr/local/host/lib64/libnvidia-ifr.so.1 to /usr/local/nvidia/lib64
2022-01-27T15:15:01.459584691+08:00 copy /usr/local/host/lib64/libnvidia-ifr.so to /usr/local/nvidia/lib64
2022-01-27T15:15:01.814857723+08:00 copy /usr/local/host/lib64/libGL.so.1 to /usr/local/nvidia/lib64
2022-01-27T15:15:01.827774087+08:00 copy /usr/local/host/lib64/libGL.so to /usr/local/nvidia/lib64
2022-01-27T15:15:01.841417973+08:00 copy /usr/local/host/lib64/libGL.so.1.7.0 to /usr/local/nvidia/lib64
2022-01-27T15:15:02.195583222+08:00 copy /usr/local/host/lib64/libGLX.so.0 to /usr/local/nvidia/lib64
2022-01-27T15:15:02.207088332+08:00 copy /usr/local/host/lib64/libGLX.so to /usr/local/nvidia/lib64
2022-01-27T15:15:02.221320790+08:00 copy /usr/local/host/lib64/libGLX.so.0.0.0 to /usr/local/nvidia/lib64
2022-01-27T15:15:02.559862624+08:00 copy /usr/local/host/lib64/libOpenGL.so.0 to /usr/local/nvidia/lib64
2022-01-27T15:15:02.572714936+08:00 copy /usr/local/host/lib64/libOpenGL.so.0.0.0 to /usr/local/nvidia/lib64
2022-01-27T15:15:02.584301511+08:00 copy /usr/local/host/lib64/libOpenGL.so to /usr/local/nvidia/lib64
2022-01-27T15:15:02.941268059+08:00 copy /usr/local/host/lib64/libGLESv1_CM.so to /usr/local/nvidia/lib64
2022-01-27T15:15:02.955333940+08:00 copy /usr/local/host/lib64/libGLESv1_CM.so.1 to /usr/local/nvidia/lib64
2022-01-27T15:15:02.967220940+08:00 copy /usr/local/host/lib64/libGLESv1_CM.so.1.2.0 to /usr/local/nvidia/lib64
2022-01-27T15:15:03.302953742+08:00 copy /usr/local/host/lib64/libGLESv2.so to /usr/local/nvidia/lib64
2022-01-27T15:15:03.314786648+08:00 copy /usr/local/host/lib64/libGLESv2.so.2 to /usr/local/nvidia/lib64
2022-01-27T15:15:03.326296602+08:00 copy /usr/local/host/lib64/libGLESv2.so.2.1.0 to /usr/local/nvidia/lib64
2022-01-27T15:15:03.666788341+08:00 copy /usr/local/host/lib64/libEGL.so.1 to /usr/local/nvidia/lib64
2022-01-27T15:15:03.679512295+08:00 copy /usr/local/host/lib64/libEGL.so to /usr/local/nvidia/lib64
2022-01-27T15:15:03.691450703+08:00 copy /usr/local/host/lib64/libEGL.so.1.1.0 to /usr/local/nvidia/lib64
2022-01-27T15:15:04.021903077+08:00 copy /usr/local/host/lib64/libGLdispatch.so.0 to /usr/local/nvidia/lib64
2022-01-27T15:15:04.032325862+08:00 copy /usr/local/host/lib64/libGLdispatch.so.0.0.0 to /usr/local/nvidia/lib64
2022-01-27T15:15:04.043387116+08:00 copy /usr/local/host/lib64/libGLdispatch.so to /usr/local/nvidia/lib64
2022-01-27T15:15:04.377466614+08:00 copy /usr/local/host/lib/libGLX_nvidia.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:15:04.389392925+08:00 copy /usr/local/host/lib/libGLX_nvidia.so.0 to /usr/local/nvidia/lib
2022-01-27T15:15:04.399746140+08:00 copy /usr/local/host/lib64/libGLX_nvidia.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:15:04.409042528+08:00 copy /usr/local/host/lib64/libGLX_nvidia.so.0 to /usr/local/nvidia/lib64
2022-01-27T15:15:04.775731657+08:00 copy /usr/local/host/lib/libEGL_nvidia.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:15:04.786955708+08:00 copy /usr/local/host/lib/libEGL_nvidia.so.0 to /usr/local/nvidia/lib
2022-01-27T15:15:04.798537412+08:00 copy /usr/local/host/lib64/libEGL_nvidia.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:15:04.809180290+08:00 copy /usr/local/host/lib64/libEGL_nvidia.so.0 to /usr/local/nvidia/lib64
2022-01-27T15:15:05.103482513+08:00 copy /usr/local/host/lib/libGLESv2_nvidia.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:15:05.117117954+08:00 copy /usr/local/host/lib/libGLESv2_nvidia.so.2 to /usr/local/nvidia/lib
2022-01-27T15:15:05.129766669+08:00 copy /usr/local/host/lib64/libGLESv2_nvidia.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:15:05.141248921+08:00 copy /usr/local/host/lib64/libGLESv2_nvidia.so.2 to /usr/local/nvidia/lib64
2022-01-27T15:15:05.495297811+08:00 copy /usr/local/host/lib/libGLESv1_CM_nvidia.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:15:05.510392919+08:00 copy /usr/local/host/lib/libGLESv1_CM_nvidia.so.1 to /usr/local/nvidia/lib
2022-01-27T15:15:05.522212773+08:00 copy /usr/local/host/lib64/libGLESv1_CM_nvidia.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:15:05.534953503+08:00 copy /usr/local/host/lib64/libGLESv1_CM_nvidia.so.1 to /usr/local/nvidia/lib64
2022-01-27T15:15:05.841311187+08:00 copy /usr/local/host/lib/libnvidia-eglcore.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:15:05.881322726+08:00 copy /usr/local/host/lib64/libnvidia-eglcore.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:15:06.252357737+08:00 copy /usr/local/host/lib64/libnvidia-egl-wayland.so.1.1.7 to /usr/local/nvidia/lib64
2022-01-27T15:15:06.262656714+08:00 copy /usr/local/host/lib64/libnvidia-egl-wayland.so.1 to /usr/local/nvidia/lib64
2022-01-27T15:15:06.628317661+08:00 copy /usr/local/host/lib/libnvidia-glcore.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:15:06.663536096+08:00 copy /usr/local/host/lib64/libnvidia-glcore.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:15:07.054514561+08:00 copy /usr/local/host/lib/libnvidia-tls.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:15:07.068586386+08:00 copy /usr/local/host/lib64/libnvidia-tls.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:15:07.429548212+08:00 copy /usr/local/host/lib/libnvidia-glsi.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:15:07.443867613+08:00 copy /usr/local/host/lib64/libnvidia-glsi.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:15:07.788040611+08:00 copy /usr/local/host/lib/libnvidia-opticalflow.so.470.82.01 to /usr/local/nvidia/lib
2022-01-27T15:15:07.800287508+08:00 copy /usr/local/host/lib/libnvidia-opticalflow.so.1 to /usr/local/nvidia/lib
2022-01-27T15:15:07.811975123+08:00 copy /usr/local/host/lib/libnvidia-opticalflow.so to /usr/local/nvidia/lib
2022-01-27T15:15:07.823496372+08:00 copy /usr/local/host/lib64/libnvidia-opticalflow.so.470.82.01 to /usr/local/nvidia/lib64
2022-01-27T15:15:07.835343610+08:00 copy /usr/local/host/lib64/libnvidia-opticalflow.so.1 to /usr/local/nvidia/lib64
2022-01-27T15:15:07.846720954+08:00 copy /usr/local/host/lib64/libnvidia-opticalflow.so to /usr/local/nvidia/lib64
2022-01-27T15:15:08.172378279+08:00 copy /usr/local/host/bin/nvidia-cuda-mps-control to /usr/local/nvidia/bin/
2022-01-27T15:15:08.513841922+08:00 copy /usr/local/host/bin/nvidia-cuda-mps-server to /usr/local/nvidia/bin/
2022-01-27T15:15:08.856701556+08:00 copy /usr/local/host/bin/nvidia-debugdump to /usr/local/nvidia/bin/
2022-01-27T15:15:09.201883209+08:00 copy /usr/local/host/bin/nvidia-persistenced to /usr/local/nvidia/bin/
2022-01-27T15:15:09.544941255+08:00 copy /usr/local/host/bin/nvidia-smi to /usr/local/nvidia/bin/
2022-01-27T15:15:09.578972619+08:00 rebuild ldcache
2022-01-27T15:15:09.589852232+08:00 launch gpu manager
2022-01-27T15:15:09.605431237+08:00 I0127 07:15:09.605194   44050 app.go:92] Wait for internal server ready
2022-01-27T15:15:09.607011470+08:00 I0127 07:15:09.606825   44050 volume.go:133] Find binaries: [/usr/local/nvidia/bin/nvidia-cuda-mps-control /usr/local/nvidia/bin/nvidia-cuda-mps-server /usr/local/nvidia/bin/nvidia-debugdump /usr/local/nvidia/bin/nvidia-persistenced /usr/local/nvidia/bin/nvidia-smi /usr/bin/gpu-client]
2022-01-27T15:15:09.609057613+08:00 I0127 07:15:09.608892   44050 volume.go:138] Find 32bit libraries: [/usr/local/nvidia/lib/libvdpau_nvidia.so.470.82.01 /usr/local/nvidia/lib/libnvidia-tls.so.470.82.01 /usr/local/nvidia/lib/libnvidia-ptxjitcompiler.so.470.82.01 /usr/local/nvidia/lib/libnvidia-opticalflow.so.470.82.01 /usr/local/nvidia/lib/libnvidia-opencl.so.470.82.01 /usr/local/nvidia/lib/libnvidia-ml.so.470.82.01 /usr/local/nvidia/lib/libnvidia-ifr.so.470.82.01 /usr/local/nvidia/lib/libnvidia-glsi.so.470.82.01 /usr/local/nvidia/lib/libnvidia-glcore.so.470.82.01 /usr/local/nvidia/lib/libnvidia-fbc.so.470.82.01 /usr/local/nvidia/lib/libnvidia-encode.so.470.82.01 /usr/local/nvidia/lib/libnvidia-eglcore.so.470.82.01 /usr/local/nvidia/lib/libnvidia-compiler.so.470.82.01 /usr/local/nvidia/lib/libnvcuvid.so.470.82.01 /usr/local/nvidia/lib/libcuda.so.470.82.01 /usr/local/nvidia/lib/libGLX_nvidia.so.470.82.01 /usr/local/nvidia/lib/libGLESv2_nvidia.so.470.82.01 /usr/local/nvidia/lib/libGLESv1_CM_nvidia.so.470.82.01 /usr/local/nvidia/lib/libEGL_nvidia.so.470.82.01]
2022-01-27T15:15:09.609081308+08:00 I0127 07:15:09.608935   44050 volume.go:139] Find 64bit libraries: [/usr/local/nvidia/lib64/libvdpau_nvidia.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-tls.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-ptxjitcompiler.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-opticalflow.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-opencl.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-ml.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-ifr.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-glsi.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-glcore.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-fbc.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-encode.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-eglcore.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-egl-wayland.so.1.1.7 /usr/local/nvidia/lib64/libnvidia-compiler.so.470.82.01 /usr/local/nvidia/lib64/libnvcuvid.so.470.82.01 /usr/local/nvidia/lib64/libcuda.so.470.82.01 /usr/lib64/libcuda-control.so /usr/local/nvidia/lib64/libOpenGL.so.0.0.0 /usr/local/nvidia/lib64/libGLdispatch.so.0.0.0 /usr/local/nvidia/lib64/libGLX_nvidia.so.470.82.01 /usr/local/nvidia/lib64/libGLX.so.0.0.0 /usr/local/nvidia/lib64/libGLESv2_nvidia.so.470.82.01 /usr/local/nvidia/lib64/libGLESv2.so.2.1.0 /usr/local/nvidia/lib64/libGLESv1_CM_nvidia.so.470.82.01 /usr/local/nvidia/lib64/libGLESv1_CM.so.1.2.0 /usr/local/nvidia/lib64/libGL.so.1.7.0 /usr/local/nvidia/lib64/libEGL_nvidia.so.470.82.01 /usr/local/nvidia/lib64/libEGL.so.1.1.0]
2022-01-27T15:15:09.610118040+08:00 I0127 07:15:09.609986   44050 volume.go:133] Find binaries: [/usr/local/nvidia/bin/nvidia-cuda-mps-control /usr/local/nvidia/bin/nvidia-cuda-mps-server /usr/local/nvidia/bin/nvidia-debugdump /usr/local/nvidia/bin/nvidia-persistenced /usr/local/nvidia/bin/nvidia-smi]
2022-01-27T15:15:09.614388448+08:00 I0127 07:15:09.614219   44050 volume.go:138] Find 32bit libraries: [/usr/local/nvidia/lib/libvdpau_nvidia.so.470.82.01 /usr/local/nvidia/lib/libnvidia-tls.so.470.82.01 /usr/local/nvidia/lib/libnvidia-ptxjitcompiler.so.470.82.01 /usr/local/nvidia/lib/libnvidia-opticalflow.so.470.82.01 /usr/local/nvidia/lib/libnvidia-opencl.so.470.82.01 /usr/local/nvidia/lib/libnvidia-ml.so.470.82.01 /usr/local/nvidia/lib/libnvidia-ifr.so.470.82.01 /usr/local/nvidia/lib/libnvidia-glsi.so.470.82.01 /usr/local/nvidia/lib/libnvidia-glcore.so.470.82.01 /usr/local/nvidia/lib/libnvidia-fbc.so.470.82.01 /usr/local/nvidia/lib/libnvidia-encode.so.470.82.01 /usr/local/nvidia/lib/libnvidia-eglcore.so.470.82.01 /usr/local/nvidia/lib/libnvidia-compiler.so.470.82.01 /usr/local/nvidia/lib/libnvcuvid.so.470.82.01 /usr/local/nvidia/lib/libcuda.so.470.82.01 /usr/local/nvidia/lib/libGLX_nvidia.so.470.82.01 /usr/local/nvidia/lib/libGLESv2_nvidia.so.470.82.01 /usr/local/nvidia/lib/libGLESv1_CM_nvidia.so.470.82.01 /usr/local/nvidia/lib/libEGL_nvidia.so.470.82.01]
2022-01-27T15:15:09.614407646+08:00 I0127 07:15:09.614264   44050 volume.go:139] Find 64bit libraries: [/usr/local/nvidia/lib64/libvdpau_nvidia.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-tls.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-ptxjitcompiler.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-opticalflow.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-opencl.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-ml.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-ifr.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-glsi.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-glcore.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-fbc.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-encode.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-eglcore.so.470.82.01 /usr/local/nvidia/lib64/libnvidia-egl-wayland.so.1.1.7 /usr/local/nvidia/lib64/libnvidia-compiler.so.470.82.01 /usr/local/nvidia/lib64/libnvcuvid.so.470.82.01 /usr/local/nvidia/lib64/libcuda.so.470.82.01 /usr/local/nvidia/lib64/libOpenGL.so.0.0.0 /usr/local/nvidia/lib64/libGLdispatch.so.0.0.0 /usr/local/nvidia/lib64/libGLX_nvidia.so.470.82.01 /usr/local/nvidia/lib64/libGLX.so.0.0.0 /usr/local/nvidia/lib64/libGLESv2_nvidia.so.470.82.01 /usr/local/nvidia/lib64/libGLESv2.so.2.1.0 /usr/local/nvidia/lib64/libGLESv1_CM_nvidia.so.470.82.01 /usr/local/nvidia/lib64/libGLESv1_CM.so.1.2.0 /usr/local/nvidia/lib64/libGL.so.1.7.0 /usr/local/nvidia/lib64/libEGL_nvidia.so.470.82.01 /usr/local/nvidia/lib64/libEGL.so.1.1.0]
2022-01-27T15:15:09.614414781+08:00 I0127 07:15:09.614309   44050 volume.go:176] Mirror /usr/local/nvidia/bin/nvidia-cuda-mps-control to /etc/gpu-manager/vdriver/nvidia/bin
2022-01-27T15:15:09.614958244+08:00 I0127 07:15:09.614857   44050 volume.go:176] Mirror /usr/local/nvidia/bin/nvidia-cuda-mps-server to /etc/gpu-manager/vdriver/nvidia/bin
2022-01-27T15:15:09.615319757+08:00 I0127 07:15:09.615222   44050 volume.go:176] Mirror /usr/local/nvidia/bin/nvidia-debugdump to /etc/gpu-manager/vdriver/nvidia/bin
2022-01-27T15:15:09.615929873+08:00 I0127 07:15:09.615841   44050 volume.go:176] Mirror /usr/local/nvidia/bin/nvidia-persistenced to /etc/gpu-manager/vdriver/nvidia/bin
2022-01-27T15:15:09.616484687+08:00 I0127 07:15:09.616400   44050 volume.go:176] Mirror /usr/local/nvidia/bin/nvidia-smi to /etc/gpu-manager/vdriver/nvidia/bin
2022-01-27T15:15:09.617562557+08:00 I0127 07:15:09.617472   44050 volume.go:176] Mirror /usr/bin/gpu-client to /etc/gpu-manager/vdriver/nvidia/bin
2022-01-27T15:15:09.638130223+08:00 I0127 07:15:09.637990   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libvdpau_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.639448607+08:00 I0127 07:15:09.639358   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-tls.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.639847885+08:00 I0127 07:15:09.639762   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-ptxjitcompiler.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.654420601+08:00 I0127 07:15:09.654257   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-opticalflow.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.654894720+08:00 I0127 07:15:09.654786   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-opencl.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.674533377+08:00 I0127 07:15:09.674368   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-ml.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.677278132+08:00 I0127 07:15:09.677145   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-ifr.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.677920586+08:00 I0127 07:15:09.677824   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-glsi.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.678857755+08:00 I0127 07:15:09.678786   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-glcore.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.711395371+08:00 I0127 07:15:09.711228   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-fbc.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.711971243+08:00 I0127 07:15:09.711874   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-encode.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.712518377+08:00 I0127 07:15:09.712389   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-eglcore.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.742380053+08:00 I0127 07:15:09.742212   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-compiler.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.805781712+08:00 I0127 07:15:09.805637   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvcuvid.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.811704362+08:00 I0127 07:15:09.811559   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libcuda.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.836848789+08:00 I0127 07:15:09.836743   44050 volume.go:185] Driver version: 470.0
2022-01-27T15:15:09.836873936+08:00 I0127 07:15:09.836761   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libGLX_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.838548410+08:00 I0127 07:15:09.838447   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libGLESv2_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.839075531+08:00 I0127 07:15:09.838947   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libGLESv1_CM_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.839516821+08:00 I0127 07:15:09.839393   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libEGL_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib
2022-01-27T15:15:09.842164547+08:00 I0127 07:15:09.841988   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libvdpau_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:09.843602999+08:00 I0127 07:15:09.843474   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-tls.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:09.843980882+08:00 I0127 07:15:09.843864   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-ptxjitcompiler.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:09.859241489+08:00 I0127 07:15:09.859094   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-opticalflow.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:09.859691405+08:00 I0127 07:15:09.859536   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-opencl.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:09.882554855+08:00 I0127 07:15:09.882382   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-ml.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:09.885252141+08:00 I0127 07:15:09.885112   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-ifr.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:09.885872805+08:00 I0127 07:15:09.885773   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-glsi.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:09.886845510+08:00 I0127 07:15:09.886723   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-glcore.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:09.920564818+08:00 I0127 07:15:09.920398   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-fbc.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:09.921103454+08:00 I0127 07:15:09.921002   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-encode.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:09.921610440+08:00 I0127 07:15:09.921474   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-eglcore.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:09.955770883+08:00 I0127 07:15:09.955584   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-egl-wayland.so.1.1.7 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:09.956240461+08:00 I0127 07:15:09.956096   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-compiler.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:10.029161286+08:00 I0127 07:15:10.029032   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvcuvid.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:10.037381521+08:00 I0127 07:15:10.037237   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libcuda.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:10.072454944+08:00 I0127 07:15:10.072296   44050 volume.go:185] Driver version: 470.0
2022-01-27T15:15:10.072479484+08:00 I0127 07:15:10.072316   44050 volume.go:176] Mirror /usr/lib64/libcuda-control.so to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:10.073931016+08:00 I0127 07:15:10.073817   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libOpenGL.so.0.0.0 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:10.074624543+08:00 I0127 07:15:10.074515   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libGLdispatch.so.0.0.0 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:10.075807915+08:00 I0127 07:15:10.075715   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libGLX_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:10.078407236+08:00 I0127 07:15:10.078327   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libGLX.so.0.0.0 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:10.080281753+08:00 I0127 07:15:10.080191   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libGLESv2_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:10.081492821+08:00 I0127 07:15:10.081363   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libGLESv2.so.2.1.0 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:10.082613821+08:00 I0127 07:15:10.082482   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libGLESv1_CM_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:10.083597334+08:00 I0127 07:15:10.083469   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libGLESv1_CM.so.1.2.0 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:10.084542206+08:00 I0127 07:15:10.084415   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libGL.so.1.7.0 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:10.086745915+08:00 I0127 07:15:10.086607   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libEGL_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:10.090284804+08:00 I0127 07:15:10.090168   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libEGL.so.1.1.0 to /etc/gpu-manager/vdriver/nvidia/lib64
2022-01-27T15:15:10.091062468+08:00 I0127 07:15:10.090970   44050 volume.go:176] Mirror /usr/local/nvidia/bin/nvidia-cuda-mps-control to /etc/gpu-manager/vdriver/origin/bin
2022-01-27T15:15:10.091526374+08:00 I0127 07:15:10.091471   44050 volume.go:176] Mirror /usr/local/nvidia/bin/nvidia-cuda-mps-server to /etc/gpu-manager/vdriver/origin/bin
2022-01-27T15:15:10.092071065+08:00 I0127 07:15:10.091941   44050 volume.go:176] Mirror /usr/local/nvidia/bin/nvidia-debugdump to /etc/gpu-manager/vdriver/origin/bin
2022-01-27T15:15:10.092743201+08:00 I0127 07:15:10.092611   44050 volume.go:176] Mirror /usr/local/nvidia/bin/nvidia-persistenced to /etc/gpu-manager/vdriver/origin/bin
2022-01-27T15:15:10.093329998+08:00 I0127 07:15:10.093244   44050 volume.go:176] Mirror /usr/local/nvidia/bin/nvidia-smi to /etc/gpu-manager/vdriver/origin/bin
2022-01-27T15:15:10.094585093+08:00 I0127 07:15:10.094501   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libvdpau_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.096084665+08:00 I0127 07:15:10.095959   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-tls.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.096466696+08:00 I0127 07:15:10.096347   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-ptxjitcompiler.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.113695447+08:00 I0127 07:15:10.113559   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-opticalflow.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.114315495+08:00 I0127 07:15:10.114248   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-opencl.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.137646435+08:00 I0127 07:15:10.137540   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-ml.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.140070290+08:00 I0127 07:15:10.139897   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-ifr.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.140649545+08:00 I0127 07:15:10.140560   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-glsi.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.141657828+08:00 I0127 07:15:10.141530   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-glcore.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.177298095+08:00 I0127 07:15:10.177150   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-fbc.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.178047202+08:00 I0127 07:15:10.177906   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-encode.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.178855251+08:00 I0127 07:15:10.178744   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-eglcore.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.227882827+08:00 I0127 07:15:10.227722   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvidia-compiler.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.311913030+08:00 I0127 07:15:10.311770   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libnvcuvid.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.318614913+08:00 I0127 07:15:10.318483   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libcuda.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.347139455+08:00 I0127 07:15:10.347009   44050 volume.go:185] Driver version: 470.0
2022-01-27T15:15:10.347165855+08:00 I0127 07:15:10.347028   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libGLX_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.348952866+08:00 I0127 07:15:10.348889   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libGLESv2_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.349464258+08:00 I0127 07:15:10.349390   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libGLESv1_CM_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.349878024+08:00 I0127 07:15:10.349825   44050 volume.go:176] Mirror /usr/local/nvidia/lib/libEGL_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib
2022-01-27T15:15:10.351781633+08:00 I0127 07:15:10.351714   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libvdpau_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.353081536+08:00 I0127 07:15:10.352996   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-tls.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.353428512+08:00 I0127 07:15:10.353306   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-ptxjitcompiler.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.367311128+08:00 I0127 07:15:10.367158   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-opticalflow.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.367729244+08:00 I0127 07:15:10.367618   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-opencl.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.391425529+08:00 I0127 07:15:10.391265   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-ml.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.394071588+08:00 I0127 07:15:10.393962   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-ifr.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.394730318+08:00 I0127 07:15:10.394617   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-glsi.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.395761756+08:00 I0127 07:15:10.395689   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-glcore.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.436364636+08:00 I0127 07:15:10.436229   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-fbc.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.437002370+08:00 I0127 07:15:10.436909   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-encode.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.437472487+08:00 I0127 07:15:10.437411   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-eglcore.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.474168277+08:00 I0127 07:15:10.474005   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-egl-wayland.so.1.1.7 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.474726464+08:00 I0127 07:15:10.474608   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvidia-compiler.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.540799094+08:00 I0127 07:15:10.540689   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libnvcuvid.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.550966010+08:00 I0127 07:15:10.550850   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libcuda.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.585864559+08:00 I0127 07:15:10.585756   44050 volume.go:185] Driver version: 470.0
2022-01-27T15:15:10.585897631+08:00 I0127 07:15:10.585778   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libOpenGL.so.0.0.0 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.586628457+08:00 I0127 07:15:10.586528   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libGLdispatch.so.0.0.0 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.587898431+08:00 I0127 07:15:10.587816   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libGLX_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.590044638+08:00 I0127 07:15:10.589945   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libGLX.so.0.0.0 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.590593784+08:00 I0127 07:15:10.590468   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libGLESv2_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.591173574+08:00 I0127 07:15:10.591037   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libGLESv2.so.2.1.0 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.591647319+08:00 I0127 07:15:10.591579   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libGLESv1_CM_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.592241121+08:00 I0127 07:15:10.592122   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libGLESv1_CM.so.1.2.0 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.592757185+08:00 I0127 07:15:10.592620   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libGL.so.1.7.0 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.593922948+08:00 I0127 07:15:10.593834   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libEGL_nvidia.so.470.82.01 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.595955171+08:00 I0127 07:15:10.595886   44050 volume.go:176] Mirror /usr/local/nvidia/lib64/libEGL.so.1.1.0 to /etc/gpu-manager/vdriver/origin/lib64
2022-01-27T15:15:10.597452250+08:00 I0127 07:15:10.597370   44050 volume.go:205] Vcuda /usr/lib64/libcuda-control.so to /etc/gpu-manager/vdriver/nvidia/lib/libcuda.so.1
2022-01-27T15:15:10.598282680+08:00 I0127 07:15:10.598225   44050 volume.go:216] Vcuda /usr/lib64/libcuda-control.so to /etc/gpu-manager/vdriver/nvidia/lib/libcuda.so
2022-01-27T15:15:10.599117899+08:00 I0127 07:15:10.599054   44050 volume.go:205] Vcuda /usr/lib64/libcuda-control.so to /etc/gpu-manager/vdriver/nvidia/lib64/libcuda.so.1
2022-01-27T15:15:10.599971725+08:00 I0127 07:15:10.599883   44050 volume.go:216] Vcuda /usr/lib64/libcuda-control.so to /etc/gpu-manager/vdriver/nvidia/lib64/libcuda.so
2022-01-27T15:15:10.600847591+08:00 I0127 07:15:10.600761   44050 volume.go:205] Vcuda /usr/lib64/libcuda-control.so to /etc/gpu-manager/vdriver/nvidia/lib/libnvidia-ml.so.1
2022-01-27T15:15:10.601695896+08:00 I0127 07:15:10.601619   44050 volume.go:216] Vcuda /usr/lib64/libcuda-control.so to /etc/gpu-manager/vdriver/nvidia/lib/libnvidia-ml.so
2022-01-27T15:15:10.602574446+08:00 I0127 07:15:10.602472   44050 volume.go:205] Vcuda /usr/lib64/libcuda-control.so to /etc/gpu-manager/vdriver/nvidia/lib64/libnvidia-ml.so.1
2022-01-27T15:15:10.603436041+08:00 I0127 07:15:10.603325   44050 volume.go:216] Vcuda /usr/lib64/libcuda-control.so to /etc/gpu-manager/vdriver/nvidia/lib64/libnvidia-ml.so
2022-01-27T15:15:10.603451301+08:00 I0127 07:15:10.603340   44050 volume.go:152] Volume manager is running
2022-01-27T15:15:10.603456583+08:00 E0127 07:15:10.603381   44050 server.go:133] Unable to set Type=notify in systemd service file?
2022-01-27T15:15:10.603470085+08:00 W0127 07:15:10.603401   44050 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2022-01-27T15:15:10.605219002+08:00 I0127 07:15:10.605101   44050 logs.go:79] parsed scheme: ""
2022-01-27T15:15:10.605233835+08:00 I0127 07:15:10.605114   44050 logs.go:79] scheme "" not registered, fallback to default scheme
2022-01-27T15:15:10.605238595+08:00 I0127 07:15:10.605123   44050 logs.go:79] ccResolverWrapper: sending update to cc: {[{/run/containerd/containerd.sock 0  <nil>}] <nil>}
2022-01-27T15:15:10.605242981+08:00 I0127 07:15:10.605147   44050 logs.go:79] ClientConn switching balancer to "pick_first"
2022-01-27T15:15:10.605276764+08:00 I0127 07:15:10.605239   44050 logs.go:79] pickfirstBalancer: HandleSubConnStateChange: 0xc000123460, CONNECTING
2022-01-27T15:15:10.605378798+08:00 I0127 07:15:10.605320   44050 app.go:92] Wait for internal server ready
2022-01-27T15:15:10.605933302+08:00 I0127 07:15:10.605861   44050 logs.go:79] pickfirstBalancer: HandleSubConnStateChange: 0xc000123460, READY
2022-01-27T15:15:10.608222773+08:00 I0127 07:15:10.608170   44050 runtime.go:69] Container runtime is containerd
2022-01-27T15:15:10.608237690+08:00 I0127 07:15:10.608202   44050 server.go:157] Container runtime manager is running
2022-01-27T15:15:10.608469937+08:00 I0127 07:15:10.608407   44050 reflector.go:150] Starting reflector *v1.Pod (1m0s) from pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105
2022-01-27T15:15:10.608490996+08:00 I0127 07:15:10.608432   44050 reflector.go:185] Listing and watching *v1.Pod from pkg/mod/k8s.io/client-go@v0.17.4/tools/cache/reflector.go:105
2022-01-27T15:15:11.605702531+08:00 I0127 07:15:11.605496   44050 app.go:92] Wait for internal server ready
2022-01-27T15:15:11.608649989+08:00 I0127 07:15:11.608505   44050 watchdog.go:64] Pod cache is running
2022-01-27T15:15:11.608694722+08:00 I0127 07:15:11.608533   44050 server.go:160] Watchdog is running
2022-01-27T15:15:11.608704756+08:00 I0127 07:15:11.608559   44050 label.go:102] Labeler for hostname 192.168.41.104
2022-01-27T15:15:11.660964248+08:00 I0127 07:15:11.660820   44050 label.go:153] Auto label is running
2022-01-27T15:15:11.660994136+08:00 I0127 07:15:11.660850   44050 server.go:167] Load container response data
2022-01-27T15:15:11.661034245+08:00 I0127 07:15:11.660957   44050 util.go:131] Try NUMA checkpoint data format
2022-01-27T15:15:11.661629051+08:00 I0127 07:15:11.661449   44050 manager.go:69] Insert f96b0b34-8379-4581-9c87-2e98637f2fc2/empty9 allocResp
2022-01-27T15:15:11.661655118+08:00 I0127 07:15:11.661521   44050 manager.go:203] Start vDevice watcher
2022-01-27T15:15:11.661702695+08:00 I0127 07:15:11.661586   44050 util.go:206] Pod tke-platform-api-b4d45f4bc-4z5ln in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661711925+08:00 I0127 07:15:11.661599   44050 util.go:206] Pod tke-monitor-controller-bb56765d-78p6v in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661719272+08:00 I0127 07:15:11.661606   44050 util.go:206] Pod tke-application-api-6db6c5f8dd-tbf2x in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661726285+08:00 I0127 07:15:11.661614   44050 util.go:206] Pod tke-logagent-controller-c6c8bcbf6-gc6z6 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661733532+08:00 I0127 07:15:11.661622   44050 util.go:206] Pod gpu-manager-daemonset-vv7zz in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.661741012+08:00 I0127 07:15:11.661630   44050 util.go:206] Pod tke-auth-api-6fcfdb454d-kv7bb in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661747859+08:00 I0127 07:15:11.661637   44050 util.go:206] Pod influxdb-0 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661801675+08:00 I0127 07:15:11.661645   44050 util.go:206] Pod tke-platform-controller-546fc4fc97-phzgj in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661835995+08:00 I0127 07:15:11.661659   44050 util.go:206] Pod tke-platform-api-b4d45f4bc-j5dbq in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661843678+08:00 I0127 07:15:11.661681   44050 util.go:206] Pod tke-registry-api-55b6f778f7-mkhwn in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661850598+08:00 I0127 07:15:11.661688   44050 util.go:206] Pod coredns-6c654f5cc6-58bz2 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.661858152+08:00 I0127 07:15:11.661694   44050 util.go:206] Pod tke-registry-controller-5bbb898bc5-hk6rw in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661864480+08:00 I0127 07:15:11.661702   44050 util.go:206] Pod tke-monitor-api-54bc7f44f6-xxcl5 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661871729+08:00 I0127 07:15:11.661710   44050 util.go:206] Pod galaxy-daemonset-tc6t6 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.661877872+08:00 I0127 07:15:11.661723   44050 util.go:206] Pod tke-auth-controller-5fbbdd9dd9-npjrd in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661883938+08:00 I0127 07:15:11.661730   44050 util.go:206] Pod tke-gateway-4gcpl in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661890038+08:00 I0127 07:15:11.661742   44050 util.go:206] Pod tke-notify-controller-85cf89b5bb-ngkqf in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661896452+08:00 I0127 07:15:11.661752   44050 util.go:206] Pod tke-business-api-77ff89ff7-s94ml in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661925284+08:00 I0127 07:15:11.661762   44050 util.go:206] Pod tke-application-api-6db6c5f8dd-l98gc in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661954530+08:00 I0127 07:15:11.661776   44050 util.go:206] Pod tke-notify-controller-85cf89b5bb-p9bbx in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661966864+08:00 I0127 07:15:11.661787   44050 util.go:206] Pod kube-proxy-26g7w in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.661977941+08:00 I0127 07:15:11.661798   44050 util.go:206] Pod tke-business-api-77ff89ff7-kwcnw in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661988461+08:00 I0127 07:15:11.661810   44050 util.go:206] Pod tke-logagent-api-6d9b6fc4-fbsqk in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.661998930+08:00 I0127 07:15:11.661817   44050 util.go:206] Pod tke-business-controller-86879f65df-rz2kv in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662010047+08:00 I0127 07:15:11.661830   44050 util.go:206] Pod tke-notify-api-df68cb499-qtnwz in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662039020+08:00 I0127 07:15:11.661837   44050 util.go:206] Pod tke-registry-api-55b6f778f7-j849x in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662075124+08:00 I0127 07:15:11.661847   44050 util.go:206] Pod tke-registry-controller-5bbb898bc5-r8p26 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662090697+08:00 I0127 07:15:11.661862   44050 util.go:206] Pod gpu-quota-admission-76cfff49b6-gs8b5 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.662098611+08:00 I0127 07:15:11.661874   44050 util.go:206] Pod tke-auth-controller-5fbbdd9dd9-qj5fx in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662105957+08:00 I0127 07:15:11.661890   44050 util.go:206] Pod kube-controller-manager-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.662138381+08:00 I0127 07:15:11.661903   44050 util.go:206] Pod tke-monitor-api-54bc7f44f6-7pl4s in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662146684+08:00 I0127 07:15:11.661918   44050 util.go:206] Pod kube-apiserver-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.662153574+08:00 I0127 07:15:11.661929   44050 util.go:206] Pod flannel-pmfcv in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.662160511+08:00 I0127 07:15:11.661942   44050 util.go:206] Pod tke-application-controller-6c769cc9cc-8v4zc in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662167361+08:00 I0127 07:15:11.661953   44050 util.go:206] Pod coredns-6c654f5cc6-wsg7s in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.662174497+08:00 I0127 07:15:11.661965   44050 util.go:206] Pod tke-monitor-controller-bb56765d-5vl2x in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662181251+08:00 I0127 07:15:11.661980   44050 util.go:206] Pod kube-scheduler-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.662188054+08:00 I0127 07:15:11.661992   44050 util.go:206] Pod etcd-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.662195121+08:00 I0127 07:15:11.662007   44050 util.go:206] Pod tke-platform-controller-546fc4fc97-sbp9l in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662201808+08:00 I0127 07:15:11.662037   44050 util.go:206] Pod tke-logagent-api-6d9b6fc4-7mg2r in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662208548+08:00 I0127 07:15:11.662052   44050 util.go:206] Pod tke-logagent-controller-c6c8bcbf6-wc7bf in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662226041+08:00 I0127 07:15:11.662066   44050 util.go:206] Pod tke-application-controller-6c769cc9cc-lbf2s in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662233368+08:00 I0127 07:15:11.662082   44050 util.go:206] Pod metrics-server-v0.3.6-794ccd69c8-8sqxb in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.662239985+08:00 I0127 07:15:11.662093   44050 util.go:206] Pod tke-notify-api-df68cb499-hzc8w in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662246948+08:00 I0127 07:15:11.662107   44050 util.go:206] Pod tke-auth-api-6fcfdb454d-srxtq in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662253681+08:00 I0127 07:15:11.662115   44050 util.go:206] Pod tke-business-controller-86879f65df-5wf94 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662635853+08:00 I0127 07:15:11.662572   44050 manager.go:232] Recover vDevice server for /etc/gpu-manager/vm/f96b0b34-8379-4581-9c87-2e98637f2fc2
2022-01-27T15:15:11.662804880+08:00 I0127 07:15:11.662745   44050 manager.go:245] Recover vDevice server for /etc/gpu-manager/vm/f96b0b34-8379-4581-9c87-2e98637f2fc2
2022-01-27T15:15:11.662839423+08:00 I0127 07:15:11.662796   44050 manager.go:199] Virtual manager is running
2022-01-27T15:15:11.662859680+08:00 I0127 07:15:11.662811   44050 manager.go:270] Starting garbage directory collector
2022-01-27T15:15:11.662909969+08:00 I0127 07:15:11.662863   44050 util.go:206] Pod tke-monitor-controller-bb56765d-5vl2x in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662918503+08:00 I0127 07:15:11.662881   44050 util.go:206] Pod tke-auth-controller-5fbbdd9dd9-qj5fx in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662924866+08:00 I0127 07:15:11.662892   44050 util.go:206] Pod kube-controller-manager-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.662937095+08:00 I0127 07:15:11.662901   44050 util.go:206] Pod tke-monitor-api-54bc7f44f6-7pl4s in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662956341+08:00 I0127 07:15:11.662894   44050 manager.go:354] Starting process vm events
2022-01-27T15:15:11.662969852+08:00 I0127 07:15:11.662908   44050 util.go:206] Pod kube-apiserver-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.662976389+08:00 I0127 07:15:11.662934   44050 util.go:206] Pod flannel-pmfcv in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.662982318+08:00 I0127 07:15:11.662949   44050 util.go:206] Pod tke-application-controller-6c769cc9cc-8v4zc in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.662988055+08:00 I0127 07:15:11.662956   44050 util.go:206] Pod coredns-6c654f5cc6-wsg7s in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.663010438+08:00 I0127 07:15:11.662964   44050 util.go:206] Pod kube-scheduler-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.663016629+08:00 I0127 07:15:11.662972   44050 util.go:206] Pod etcd-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.663035716+08:00 I0127 07:15:11.662980   44050 util.go:206] Pod tke-business-controller-86879f65df-5wf94 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663071806+08:00 I0127 07:15:11.662987   44050 util.go:206] Pod tke-platform-controller-546fc4fc97-sbp9l in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663081499+08:00 I0127 07:15:11.662994   44050 util.go:206] Pod tke-logagent-api-6d9b6fc4-7mg2r in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663115482+08:00 I0127 07:15:11.663001   44050 util.go:206] Pod tke-logagent-controller-c6c8bcbf6-wc7bf in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663154719+08:00 I0127 07:15:11.663012   44050 util.go:206] Pod tke-application-controller-6c769cc9cc-lbf2s in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663164349+08:00 I0127 07:15:11.663022   44050 util.go:206] Pod metrics-server-v0.3.6-794ccd69c8-8sqxb in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.663172519+08:00 I0127 07:15:11.663029   44050 util.go:206] Pod tke-notify-api-df68cb499-hzc8w in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663180189+08:00 I0127 07:15:11.663036   44050 util.go:206] Pod tke-auth-api-6fcfdb454d-srxtq in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663187556+08:00 I0127 07:15:11.663043   44050 util.go:206] Pod tke-platform-api-b4d45f4bc-4z5ln in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663194640+08:00 I0127 07:15:11.663051   44050 util.go:206] Pod tke-monitor-controller-bb56765d-78p6v in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663202110+08:00 I0127 07:15:11.663058   44050 util.go:206] Pod tke-application-api-6db6c5f8dd-tbf2x in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663209243+08:00 I0127 07:15:11.663066   44050 util.go:206] Pod tke-logagent-controller-c6c8bcbf6-gc6z6 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663215916+08:00 I0127 07:15:11.663078   44050 util.go:206] Pod gpu-manager-daemonset-vv7zz in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.663222833+08:00 I0127 07:15:11.663085   44050 util.go:206] Pod tke-auth-api-6fcfdb454d-kv7bb in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663243787+08:00 I0127 07:15:11.663101   44050 util.go:206] Pod galaxy-daemonset-tc6t6 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.663264773+08:00 I0127 07:15:11.663110   44050 util.go:206] Pod influxdb-0 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663284400+08:00 I0127 07:15:11.663119   44050 util.go:206] Pod tke-platform-controller-546fc4fc97-phzgj in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663291598+08:00 I0127 07:15:11.663126   44050 util.go:206] Pod tke-platform-api-b4d45f4bc-j5dbq in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663319659+08:00 I0127 07:15:11.663135   44050 util.go:206] Pod tke-registry-api-55b6f778f7-mkhwn in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663332158+08:00 I0127 07:15:11.663147   44050 util.go:206] Pod coredns-6c654f5cc6-58bz2 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.663342038+08:00 I0127 07:15:11.663154   44050 util.go:206] Pod tke-registry-controller-5bbb898bc5-hk6rw in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663352473+08:00 I0127 07:15:11.663161   44050 util.go:206] Pod tke-monitor-api-54bc7f44f6-xxcl5 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663360622+08:00 I0127 07:15:11.663174   44050 util.go:206] Pod tke-auth-controller-5fbbdd9dd9-npjrd in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663367115+08:00 I0127 07:15:11.663181   44050 util.go:206] Pod tke-gateway-4gcpl in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663373616+08:00 I0127 07:15:11.663190   44050 util.go:206] Pod tke-notify-controller-85cf89b5bb-ngkqf in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663380016+08:00 I0127 07:15:11.663199   44050 util.go:206] Pod tke-business-api-77ff89ff7-s94ml in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663386281+08:00 I0127 07:15:11.663206   44050 util.go:206] Pod tke-application-api-6db6c5f8dd-l98gc in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663393105+08:00 I0127 07:15:11.663214   44050 util.go:206] Pod tke-notify-controller-85cf89b5bb-p9bbx in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663399539+08:00 I0127 07:15:11.663222   44050 util.go:206] Pod kube-proxy-26g7w in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.663417302+08:00 I0127 07:15:11.663229   44050 util.go:206] Pod tke-business-api-77ff89ff7-kwcnw in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663424294+08:00 I0127 07:15:11.663236   44050 util.go:206] Pod tke-logagent-api-6d9b6fc4-fbsqk in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663430748+08:00 I0127 07:15:11.663247   44050 util.go:206] Pod tke-business-controller-86879f65df-rz2kv in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663437001+08:00 I0127 07:15:11.663256   44050 util.go:206] Pod tke-notify-api-df68cb499-qtnwz in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663443414+08:00 I0127 07:15:11.663263   44050 util.go:206] Pod tke-registry-api-55b6f778f7-j849x in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663449854+08:00 I0127 07:15:11.663270   44050 util.go:206] Pod tke-registry-controller-5bbb898bc5-r8p26 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.663456228+08:00 I0127 07:15:11.663277   44050 util.go:206] Pod gpu-quota-admission-76cfff49b6-gs8b5 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.668414267+08:00 I0127 07:15:11.668300   44050 tree.go:182] Detect 1 gpu cards
2022-01-27T15:15:11.674361124+08:00 E0127 07:15:11.674269   44050 tree.go:333] No topology level found at 0
2022-01-27T15:15:11.674375390+08:00 I0127 07:15:11.674291   44050 tree.go:336] Only one card topology
2022-01-27T15:15:11.676188533+08:00 I0127 07:15:11.676107   44050 tree.go:119] Update device information
2022-01-27T15:15:11.683856601+08:00 I0127 07:15:11.683735   44050 tree.go:135] node 0, pid: [], memory: 0, utilization: 0, pendingReset: false
2022-01-27T15:15:11.684440017+08:00 I0127 07:15:11.684324   44050 allocator.go:275] Load extra config from /etc/gpu-manager/extra-config.json
2022-01-27T15:15:11.684782237+08:00 I0127 07:15:11.684713   44050 allocator.go:196] Nvidia GPU "/dev/nvidia0" is in use by container: "empty9"
2022-01-27T15:15:11.684795359+08:00 I0127 07:15:11.684733   44050 allocator.go:197] Uid: f96b0b34-8379-4581-9c87-2e98637f2fc2, Name: empty9, util: 20, memory: 805306368
2022-01-27T15:15:11.684819400+08:00 I0127 07:15:11.684789   44050 tree.go:487] Occupy /dev/nvidia0 with 20 805306368, mask 1
2022-01-27T15:15:11.684826375+08:00 I0127 07:15:11.684802   44050 tree.go:514] Occupy /dev/nvidia0 parent 1
2022-01-27T15:15:11.684840315+08:00 I0127 07:15:11.684814   44050 tree.go:497] /dev/nvidia0 cores 100->80
2022-01-27T15:15:11.684857017+08:00 I0127 07:15:11.684825   44050 tree.go:503] /dev/nvidia0 memory 16911040512->16105734144
2022-01-27T15:15:11.684903638+08:00 I0127 07:15:11.684855   44050 util.go:206] Pod tke-monitor-api-54bc7f44f6-xxcl5 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.684913151+08:00 I0127 07:15:11.684864   44050 util.go:206] Pod galaxy-daemonset-tc6t6 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.684919133+08:00 I0127 07:15:11.684870   44050 util.go:206] Pod influxdb-0 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.684924824+08:00 I0127 07:15:11.684877   44050 util.go:206] Pod tke-platform-controller-546fc4fc97-phzgj in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.684945061+08:00 I0127 07:15:11.684884   44050 util.go:206] Pod tke-platform-api-b4d45f4bc-j5dbq in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.684968501+08:00 I0127 07:15:11.684896   44050 util.go:206] Pod tke-registry-api-55b6f778f7-mkhwn in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.684975959+08:00 I0127 07:15:11.684903   44050 util.go:206] Pod coredns-6c654f5cc6-58bz2 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.684982507+08:00 I0127 07:15:11.684911   44050 util.go:206] Pod tke-registry-controller-5bbb898bc5-hk6rw in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.684997910+08:00 I0127 07:15:11.684918   44050 util.go:206] Pod tke-auth-controller-5fbbdd9dd9-npjrd in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685004852+08:00 I0127 07:15:11.684929   44050 util.go:206] Pod tke-gateway-4gcpl in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685010984+08:00 I0127 07:15:11.684939   44050 util.go:206] Pod tke-notify-controller-85cf89b5bb-ngkqf in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685017044+08:00 I0127 07:15:11.684946   44050 util.go:206] Pod tke-business-api-77ff89ff7-s94ml in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685023050+08:00 I0127 07:15:11.684952   44050 util.go:206] Pod tke-application-api-6db6c5f8dd-l98gc in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685029064+08:00 I0127 07:15:11.684959   44050 util.go:206] Pod tke-notify-controller-85cf89b5bb-p9bbx in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685038433+08:00 I0127 07:15:11.684965   44050 util.go:206] Pod kube-proxy-26g7w in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.685044350+08:00 I0127 07:15:11.684972   44050 util.go:206] Pod gpu-quota-admission-76cfff49b6-gs8b5 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.685066677+08:00 I0127 07:15:11.684989   44050 util.go:206] Pod tke-business-api-77ff89ff7-kwcnw in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685092317+08:00 I0127 07:15:11.684996   44050 util.go:206] Pod tke-logagent-api-6d9b6fc4-fbsqk in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685132083+08:00 I0127 07:15:11.685003   44050 util.go:206] Pod tke-business-controller-86879f65df-rz2kv in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685140989+08:00 I0127 07:15:11.685014   44050 util.go:206] Pod tke-notify-api-df68cb499-qtnwz in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685147726+08:00 I0127 07:15:11.685020   44050 util.go:206] Pod tke-registry-api-55b6f778f7-j849x in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685154238+08:00 I0127 07:15:11.685026   44050 util.go:206] Pod tke-registry-controller-5bbb898bc5-r8p26 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685160552+08:00 I0127 07:15:11.685035   44050 util.go:206] Pod coredns-6c654f5cc6-wsg7s in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.685166933+08:00 I0127 07:15:11.685042   44050 util.go:206] Pod tke-monitor-controller-bb56765d-5vl2x in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685185902+08:00 I0127 07:15:11.685048   44050 util.go:206] Pod tke-auth-controller-5fbbdd9dd9-qj5fx in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685199160+08:00 I0127 07:15:11.685054   44050 util.go:206] Pod kube-controller-manager-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.685205633+08:00 I0127 07:15:11.685061   44050 util.go:206] Pod tke-monitor-api-54bc7f44f6-7pl4s in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685211613+08:00 I0127 07:15:11.685067   44050 util.go:206] Pod kube-apiserver-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.685217763+08:00 I0127 07:15:11.685074   44050 util.go:206] Pod flannel-pmfcv in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.685223763+08:00 I0127 07:15:11.685081   44050 util.go:206] Pod tke-application-controller-6c769cc9cc-8v4zc in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685229745+08:00 I0127 07:15:11.685087   44050 util.go:206] Pod kube-scheduler-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.685235545+08:00 I0127 07:15:11.685093   44050 util.go:206] Pod etcd-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.685241385+08:00 I0127 07:15:11.685099   44050 util.go:206] Pod tke-auth-api-6fcfdb454d-srxtq in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685247156+08:00 I0127 07:15:11.685106   44050 util.go:206] Pod tke-business-controller-86879f65df-5wf94 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685252916+08:00 I0127 07:15:11.685113   44050 util.go:206] Pod tke-platform-controller-546fc4fc97-sbp9l in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685283654+08:00 I0127 07:15:11.685120   44050 util.go:206] Pod tke-logagent-api-6d9b6fc4-7mg2r in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685296579+08:00 I0127 07:15:11.685126   44050 util.go:206] Pod tke-logagent-controller-c6c8bcbf6-wc7bf in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685303025+08:00 I0127 07:15:11.685133   44050 util.go:206] Pod tke-application-controller-6c769cc9cc-lbf2s in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685309428+08:00 I0127 07:15:11.685140   44050 util.go:206] Pod metrics-server-v0.3.6-794ccd69c8-8sqxb in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.685316079+08:00 I0127 07:15:11.685147   44050 util.go:206] Pod tke-notify-api-df68cb499-hzc8w in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685327993+08:00 I0127 07:15:11.685153   44050 util.go:206] Pod tke-platform-api-b4d45f4bc-4z5ln in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685335556+08:00 I0127 07:15:11.685160   44050 util.go:206] Pod tke-monitor-controller-bb56765d-78p6v in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685353964+08:00 I0127 07:15:11.685167   44050 util.go:206] Pod tke-application-api-6db6c5f8dd-tbf2x in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685360751+08:00 I0127 07:15:11.685178   44050 util.go:206] Pod tke-logagent-controller-c6c8bcbf6-gc6z6 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685367548+08:00 I0127 07:15:11.685184   44050 util.go:206] Pod gpu-manager-daemonset-vv7zz in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.685373876+08:00 I0127 07:15:11.685191   44050 util.go:206] Pod tke-auth-api-6fcfdb454d-kv7bb in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.685380527+08:00 I0127 07:15:11.685217   44050 allocator.go:631] Pods to be removed: []
2022-01-27T15:15:11.845831500+08:00 I0127 07:15:11.845712   44050 allocator.go:216] Checking allocation of pods on this node
2022-01-27T15:15:11.935165188+08:00 I0127 07:15:11.935015   44050 util.go:216] Determine if the container empty9 needs GPU resource
2022-01-27T15:15:11.935189368+08:00 I0127 07:15:11.935053   44050 util.go:206] Pod coredns-6c654f5cc6-58bz2 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.935196871+08:00 I0127 07:15:11.935063   44050 util.go:206] Pod coredns-6c654f5cc6-wsg7s in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.935203689+08:00 I0127 07:15:11.935070   44050 util.go:206] Pod etcd-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.935209691+08:00 I0127 07:15:11.935076   44050 util.go:206] Pod flannel-pmfcv in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.935216094+08:00 I0127 07:15:11.935082   44050 util.go:206] Pod galaxy-daemonset-tc6t6 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.935225360+08:00 I0127 07:15:11.935089   44050 util.go:206] Pod gpu-manager-daemonset-vv7zz in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.935231614+08:00 I0127 07:15:11.935095   44050 util.go:206] Pod gpu-quota-admission-76cfff49b6-gs8b5 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.935237706+08:00 I0127 07:15:11.935102   44050 util.go:206] Pod kube-apiserver-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.935243666+08:00 I0127 07:15:11.935108   44050 util.go:206] Pod kube-controller-manager-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.935249723+08:00 I0127 07:15:11.935115   44050 util.go:206] Pod kube-proxy-26g7w in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.935255963+08:00 I0127 07:15:11.935121   44050 util.go:206] Pod kube-scheduler-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.935261969+08:00 I0127 07:15:11.935128   44050 util.go:206] Pod metrics-server-v0.3.6-794ccd69c8-8sqxb in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:11.935268138+08:00 I0127 07:15:11.935135   44050 util.go:206] Pod influxdb-0 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935288864+08:00 I0127 07:15:11.935141   44050 util.go:206] Pod tke-application-api-6db6c5f8dd-l98gc in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935295526+08:00 I0127 07:15:11.935147   44050 util.go:206] Pod tke-application-api-6db6c5f8dd-tbf2x in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935311675+08:00 I0127 07:15:11.935154   44050 util.go:206] Pod tke-application-controller-6c769cc9cc-8v4zc in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935318455+08:00 I0127 07:15:11.935160   44050 util.go:206] Pod tke-application-controller-6c769cc9cc-lbf2s in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935324315+08:00 I0127 07:15:11.935167   44050 util.go:206] Pod tke-auth-api-6fcfdb454d-kv7bb in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935330158+08:00 I0127 07:15:11.935174   44050 util.go:206] Pod tke-auth-api-6fcfdb454d-srxtq in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935335984+08:00 I0127 07:15:11.935182   44050 util.go:206] Pod tke-auth-controller-5fbbdd9dd9-npjrd in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935341884+08:00 I0127 07:15:11.935189   44050 util.go:206] Pod tke-auth-controller-5fbbdd9dd9-qj5fx in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935347967+08:00 I0127 07:15:11.935196   44050 util.go:206] Pod tke-business-api-77ff89ff7-kwcnw in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935353692+08:00 I0127 07:15:11.935204   44050 util.go:206] Pod tke-business-api-77ff89ff7-s94ml in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935359744+08:00 I0127 07:15:11.935211   44050 util.go:206] Pod tke-business-controller-86879f65df-5wf94 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935365547+08:00 I0127 07:15:11.935219   44050 util.go:206] Pod tke-business-controller-86879f65df-rz2kv in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935372301+08:00 I0127 07:15:11.935225   44050 util.go:206] Pod tke-gateway-4gcpl in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935396787+08:00 I0127 07:15:11.935232   44050 util.go:206] Pod tke-logagent-api-6d9b6fc4-7mg2r in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935403179+08:00 I0127 07:15:11.935238   44050 util.go:206] Pod tke-logagent-api-6d9b6fc4-fbsqk in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935408870+08:00 I0127 07:15:11.935245   44050 util.go:206] Pod tke-logagent-controller-c6c8bcbf6-gc6z6 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935414884+08:00 I0127 07:15:11.935253   44050 util.go:206] Pod tke-logagent-controller-c6c8bcbf6-wc7bf in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935420732+08:00 I0127 07:15:11.935260   44050 util.go:206] Pod tke-monitor-api-54bc7f44f6-7pl4s in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935426492+08:00 I0127 07:15:11.935267   44050 util.go:206] Pod tke-monitor-api-54bc7f44f6-xxcl5 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935432262+08:00 I0127 07:15:11.935274   44050 util.go:206] Pod tke-monitor-controller-bb56765d-5vl2x in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935448064+08:00 I0127 07:15:11.935280   44050 util.go:206] Pod tke-monitor-controller-bb56765d-78p6v in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935454479+08:00 I0127 07:15:11.935287   44050 util.go:206] Pod tke-notify-api-df68cb499-hzc8w in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935460475+08:00 I0127 07:15:11.935293   44050 util.go:206] Pod tke-notify-api-df68cb499-qtnwz in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935466224+08:00 I0127 07:15:11.935300   44050 util.go:206] Pod tke-notify-controller-85cf89b5bb-ngkqf in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935471947+08:00 I0127 07:15:11.935306   44050 util.go:206] Pod tke-notify-controller-85cf89b5bb-p9bbx in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935481562+08:00 I0127 07:15:11.935313   44050 util.go:206] Pod tke-platform-api-b4d45f4bc-4z5ln in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935488010+08:00 I0127 07:15:11.935319   44050 util.go:206] Pod tke-platform-api-b4d45f4bc-j5dbq in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935493987+08:00 I0127 07:15:11.935326   44050 util.go:206] Pod tke-platform-controller-546fc4fc97-phzgj in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935499802+08:00 I0127 07:15:11.935333   44050 util.go:206] Pod tke-platform-controller-546fc4fc97-sbp9l in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935505787+08:00 I0127 07:15:11.935340   44050 util.go:206] Pod tke-registry-api-55b6f778f7-j849x in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935511527+08:00 I0127 07:15:11.935347   44050 util.go:206] Pod tke-registry-api-55b6f778f7-mkhwn in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935517202+08:00 I0127 07:15:11.935354   44050 util.go:206] Pod tke-registry-controller-5bbb898bc5-hk6rw in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935522927+08:00 I0127 07:15:11.935360   44050 util.go:206] Pod tke-registry-controller-5bbb898bc5-r8p26 in namespace tke does not Request for GPU resource
2022-01-27T15:15:11.935537453+08:00 I0127 07:15:11.935384   44050 server.go:191] Starting the GRPC server, driver nvidia, queryPort 5678
2022-01-27T15:15:11.935592763+08:00 I0127 07:15:11.935513   44050 server.go:245] Server tencent.com/vcuda-core is running
2022-01-27T15:15:11.935631370+08:00 I0127 07:15:11.935529   44050 server.go:245] Server tencent.com/vcuda-memory is running
2022-01-27T15:15:11.935654220+08:00 I0127 07:15:11.935619   44050 server.go:259] Server is ready at /var/run/gpu-manager.sock
2022-01-27T15:15:11.935736299+08:00 I0127 07:15:11.935634   44050 logs.go:79] parsed scheme: ""
2022-01-27T15:15:11.935766376+08:00 I0127 07:15:11.935652   44050 logs.go:79] scheme "" not registered, fallback to default scheme
2022-01-27T15:15:11.935796624+08:00 I0127 07:15:11.935685   44050 logs.go:79] ccResolverWrapper: sending update to cc: {[{/var/run/gpu-manager.sock 0  <nil>}] <nil>}
2022-01-27T15:15:11.935808699+08:00 I0127 07:15:11.935721   44050 logs.go:79] ClientConn switching balancer to "pick_first"
2022-01-27T15:15:11.935891561+08:00 I0127 07:15:11.935829   44050 logs.go:79] pickfirstBalancer: HandleSubConnStateChange: 0xc0006dc160, CONNECTING
2022-01-27T15:15:11.935918288+08:00 I0127 07:15:11.935864   44050 vcore.go:82] Server tencent.com/vcuda-core is ready at /var/lib/kubelet/device-plugins/vcore.sock
2022-01-27T15:15:11.935980500+08:00 I0127 07:15:11.935910   44050 vmemory.go:81] Server tencent.com/vcuda-memory is ready at /var/lib/kubelet/device-plugins/vmemory.sock
2022-01-27T15:15:11.936195686+08:00 I0127 07:15:11.936133   44050 logs.go:79] pickfirstBalancer: HandleSubConnStateChange: 0xc0006dc160, READY
2022-01-27T15:15:12.605930673+08:00 I0127 07:15:12.605740   44050 logs.go:79] parsed scheme: ""
2022-01-27T15:15:12.605962556+08:00 I0127 07:15:12.605775   44050 logs.go:79] scheme "" not registered, fallback to default scheme
2022-01-27T15:15:12.605973233+08:00 I0127 07:15:12.605791   44050 logs.go:79] ccResolverWrapper: sending update to cc: {[{/var/lib/kubelet/device-plugins/vcore.sock 0  <nil>}] <nil>}
2022-01-27T15:15:12.605981086+08:00 I0127 07:15:12.605821   44050 logs.go:79] ClientConn switching balancer to "pick_first"
2022-01-27T15:15:12.606003403+08:00 I0127 07:15:12.605937   44050 logs.go:79] pickfirstBalancer: HandleSubConnStateChange: 0xc0000dd290, CONNECTING
2022-01-27T15:15:12.606505102+08:00 I0127 07:15:12.606346   44050 logs.go:79] pickfirstBalancer: HandleSubConnStateChange: 0xc0000dd290, READY
2022-01-27T15:15:12.606542916+08:00 I0127 07:15:12.606383   44050 server.go:92] Server /var/lib/kubelet/device-plugins/vcore.sock is ready, readyServers: 1
2022-01-27T15:15:12.606552794+08:00 I0127 07:15:12.606401   44050 logs.go:79] parsed scheme: ""
2022-01-27T15:15:12.606559828+08:00 I0127 07:15:12.606410   44050 logs.go:79] scheme "" not registered, fallback to default scheme
2022-01-27T15:15:12.606566773+08:00 I0127 07:15:12.606422   44050 logs.go:79] ccResolverWrapper: sending update to cc: {[{/var/lib/kubelet/device-plugins/vmemory.sock 0  <nil>}] <nil>}
2022-01-27T15:15:12.606573514+08:00 I0127 07:15:12.606440   44050 logs.go:79] ClientConn switching balancer to "pick_first"
2022-01-27T15:15:12.606590551+08:00 I0127 07:15:12.606443   44050 logs.go:79] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2022-01-27T15:15:12.606598263+08:00 I0127 07:15:12.606503   44050 logs.go:79] pickfirstBalancer: HandleSubConnStateChange: 0xc0000dd4d0, CONNECTING
2022-01-27T15:15:12.606686133+08:00 I0127 07:15:12.606588   44050 logs.go:79] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2022-01-27T15:15:12.607031082+08:00 I0127 07:15:12.606933   44050 logs.go:79] pickfirstBalancer: HandleSubConnStateChange: 0xc0000dd4d0, READY
2022-01-27T15:15:12.607050519+08:00 I0127 07:15:12.606966   44050 server.go:92] Server /var/lib/kubelet/device-plugins/vmemory.sock is ready, readyServers: 2
2022-01-27T15:15:12.607063302+08:00 I0127 07:15:12.606995   44050 logs.go:79] parsed scheme: ""
2022-01-27T15:15:12.607075405+08:00 I0127 07:15:12.607003   44050 logs.go:79] scheme "" not registered, fallback to default scheme
2022-01-27T15:15:12.607108937+08:00 I0127 07:15:12.607013   44050 logs.go:79] ccResolverWrapper: sending update to cc: {[{/var/lib/kubelet/device-plugins/kubelet.sock 0  <nil>}] <nil>}
2022-01-27T15:15:12.607134556+08:00 I0127 07:15:12.607028   44050 logs.go:79] ClientConn switching balancer to "pick_first"
2022-01-27T15:15:12.607154116+08:00 I0127 07:15:12.607059   44050 logs.go:79] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2022-01-27T15:15:12.607193453+08:00 I0127 07:15:12.607114   44050 logs.go:79] pickfirstBalancer: HandleSubConnStateChange: 0xc00098a1b0, CONNECTING
2022-01-27T15:15:12.607207313+08:00 I0127 07:15:12.607135   44050 logs.go:79] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2022-01-27T15:15:12.607641528+08:00 I0127 07:15:12.607531   44050 logs.go:79] pickfirstBalancer: HandleSubConnStateChange: 0xc00098a1b0, READY
2022-01-27T15:15:12.607692309+08:00 I0127 07:15:12.607566   44050 server.go:343] Register to kubelet with endpoint vcore.sock
2022-01-27T15:15:12.608429472+08:00 I0127 07:15:12.608330   44050 server.go:343] Register to kubelet with endpoint vmemory.sock
2022-01-27T15:15:12.609061739+08:00 I0127 07:15:12.608933   44050 vcore.go:94] ListAndWatch request for vcore
2022-01-27T15:15:12.609099686+08:00 I0127 07:15:12.609011   44050 logs.go:79] transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2022-01-27T15:15:12.609675039+08:00 I0127 07:15:12.609515   44050 vmemory.go:98] ListAndWatch request for vmemory

2022-01-27T15:15:41.935937280+08:00 I0127 07:15:41.935722   44050 allocator.go:216] Checking allocation of pods on this node
2022-01-27T15:15:41.992878184+08:00 I0127 07:15:41.992695   44050 util.go:216] Determine if the container empty9 needs GPU resource
2022-01-27T15:15:41.992912047+08:00 I0127 07:15:41.992742   44050 util.go:206] Pod coredns-6c654f5cc6-58bz2 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:41.992921351+08:00 I0127 07:15:41.992751   44050 util.go:206] Pod coredns-6c654f5cc6-wsg7s in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:41.992938301+08:00 I0127 07:15:41.992758   44050 util.go:206] Pod etcd-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:41.992947464+08:00 I0127 07:15:41.992764   44050 util.go:206] Pod flannel-pmfcv in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:41.992954514+08:00 I0127 07:15:41.992771   44050 util.go:206] Pod galaxy-daemonset-tc6t6 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:41.992961351+08:00 I0127 07:15:41.992777   44050 util.go:206] Pod gpu-manager-daemonset-vv7zz in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:41.992968861+08:00 I0127 07:15:41.992783   44050 util.go:206] Pod gpu-quota-admission-76cfff49b6-gs8b5 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:41.992976224+08:00 I0127 07:15:41.992790   44050 util.go:206] Pod kube-apiserver-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:41.992983728+08:00 I0127 07:15:41.992795   44050 util.go:206] Pod kube-controller-manager-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:41.992990651+08:00 I0127 07:15:41.992802   44050 util.go:206] Pod kube-proxy-26g7w in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:41.992997654+08:00 I0127 07:15:41.992808   44050 util.go:206] Pod kube-scheduler-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:41.993004478+08:00 I0127 07:15:41.992817   44050 util.go:206] Pod metrics-server-v0.3.6-794ccd69c8-8sqxb in namespace kube-system does not Request for GPU resource
2022-01-27T15:15:41.993013738+08:00 I0127 07:15:41.992823   44050 util.go:206] Pod influxdb-0 in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993021311+08:00 I0127 07:15:41.992830   44050 util.go:206] Pod tke-application-api-6db6c5f8dd-l98gc in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993028378+08:00 I0127 07:15:41.992837   44050 util.go:206] Pod tke-application-api-6db6c5f8dd-tbf2x in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993035424+08:00 I0127 07:15:41.992844   44050 util.go:206] Pod tke-application-controller-6c769cc9cc-8v4zc in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993070969+08:00 I0127 07:15:41.992851   44050 util.go:206] Pod tke-application-controller-6c769cc9cc-lbf2s in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993108709+08:00 I0127 07:15:41.992857   44050 util.go:206] Pod tke-auth-api-6fcfdb454d-kv7bb in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993116041+08:00 I0127 07:15:41.992863   44050 util.go:206] Pod tke-auth-api-6fcfdb454d-srxtq in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993122619+08:00 I0127 07:15:41.992869   44050 util.go:206] Pod tke-auth-controller-5fbbdd9dd9-npjrd in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993128795+08:00 I0127 07:15:41.992875   44050 util.go:206] Pod tke-auth-controller-5fbbdd9dd9-qj5fx in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993135224+08:00 I0127 07:15:41.992881   44050 util.go:206] Pod tke-business-api-77ff89ff7-kwcnw in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993141647+08:00 I0127 07:15:41.992887   44050 util.go:206] Pod tke-business-api-77ff89ff7-s94ml in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993147719+08:00 I0127 07:15:41.992901   44050 util.go:206] Pod tke-business-controller-86879f65df-5wf94 in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993153532+08:00 I0127 07:15:41.992907   44050 util.go:206] Pod tke-business-controller-86879f65df-rz2kv in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993170450+08:00 I0127 07:15:41.992914   44050 util.go:206] Pod tke-gateway-4gcpl in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993177115+08:00 I0127 07:15:41.992919   44050 util.go:206] Pod tke-logagent-api-6d9b6fc4-7mg2r in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993183119+08:00 I0127 07:15:41.992926   44050 util.go:206] Pod tke-logagent-api-6d9b6fc4-fbsqk in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993205924+08:00 I0127 07:15:41.992934   44050 util.go:206] Pod tke-logagent-controller-c6c8bcbf6-gc6z6 in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993212827+08:00 I0127 07:15:41.992940   44050 util.go:206] Pod tke-logagent-controller-c6c8bcbf6-wc7bf in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993218964+08:00 I0127 07:15:41.992950   44050 util.go:206] Pod tke-monitor-api-54bc7f44f6-7pl4s in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993224793+08:00 I0127 07:15:41.992956   44050 util.go:206] Pod tke-monitor-api-54bc7f44f6-xxcl5 in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993230635+08:00 I0127 07:15:41.992962   44050 util.go:206] Pod tke-monitor-controller-bb56765d-5vl2x in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993236493+08:00 I0127 07:15:41.992968   44050 util.go:206] Pod tke-monitor-controller-bb56765d-78p6v in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993242225+08:00 I0127 07:15:41.992974   44050 util.go:206] Pod tke-notify-api-df68cb499-hzc8w in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993248247+08:00 I0127 07:15:41.992979   44050 util.go:206] Pod tke-notify-api-df68cb499-qtnwz in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993253927+08:00 I0127 07:15:41.992985   44050 util.go:206] Pod tke-notify-controller-85cf89b5bb-ngkqf in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993259602+08:00 I0127 07:15:41.992991   44050 util.go:206] Pod tke-notify-controller-85cf89b5bb-p9bbx in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993265333+08:00 I0127 07:15:41.992997   44050 util.go:206] Pod tke-platform-api-b4d45f4bc-4z5ln in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993271027+08:00 I0127 07:15:41.993003   44050 util.go:206] Pod tke-platform-api-b4d45f4bc-j5dbq in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993276750+08:00 I0127 07:15:41.993010   44050 util.go:206] Pod tke-platform-controller-546fc4fc97-phzgj in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993282407+08:00 I0127 07:15:41.993017   44050 util.go:206] Pod tke-platform-controller-546fc4fc97-sbp9l in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993288039+08:00 I0127 07:15:41.993023   44050 util.go:206] Pod tke-registry-api-55b6f778f7-j849x in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993293773+08:00 I0127 07:15:41.993029   44050 util.go:206] Pod tke-registry-api-55b6f778f7-mkhwn in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993299445+08:00 I0127 07:15:41.993035   44050 util.go:206] Pod tke-registry-controller-5bbb898bc5-hk6rw in namespace tke does not Request for GPU resource
2022-01-27T15:15:41.993305108+08:00 I0127 07:15:41.993041   44050 util.go:206] Pod tke-registry-controller-5bbb898bc5-r8p26 in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663647683+08:00 I0127 07:16:11.663480   44050 util.go:206] Pod tke-auth-api-6fcfdb454d-kv7bb in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663713963+08:00 I0127 07:16:11.663517   44050 util.go:206] Pod influxdb-0 in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663734892+08:00 I0127 07:16:11.663526   44050 util.go:206] Pod tke-platform-controller-546fc4fc97-phzgj in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663742486+08:00 I0127 07:16:11.663533   44050 util.go:206] Pod tke-platform-api-b4d45f4bc-j5dbq in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663748972+08:00 I0127 07:16:11.663542   44050 util.go:206] Pod tke-registry-api-55b6f778f7-mkhwn in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663755620+08:00 I0127 07:16:11.663550   44050 util.go:206] Pod coredns-6c654f5cc6-58bz2 in namespace kube-system does not Request for GPU resource
2022-01-27T15:16:11.663762140+08:00 I0127 07:16:11.663558   44050 util.go:206] Pod tke-registry-controller-5bbb898bc5-hk6rw in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663768231+08:00 I0127 07:16:11.663565   44050 util.go:206] Pod tke-monitor-api-54bc7f44f6-xxcl5 in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663774708+08:00 I0127 07:16:11.663572   44050 util.go:206] Pod galaxy-daemonset-tc6t6 in namespace kube-system does not Request for GPU resource
2022-01-27T15:16:11.663780721+08:00 I0127 07:16:11.663579   44050 util.go:206] Pod tke-auth-controller-5fbbdd9dd9-npjrd in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663801148+08:00 I0127 07:16:11.663586   44050 util.go:206] Pod tke-gateway-4gcpl in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663829214+08:00 I0127 07:16:11.663598   44050 util.go:206] Pod tke-notify-controller-85cf89b5bb-ngkqf in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663864451+08:00 I0127 07:16:11.663612   44050 util.go:206] Pod tke-business-api-77ff89ff7-s94ml in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663872879+08:00 I0127 07:16:11.663620   44050 util.go:206] Pod tke-application-api-6db6c5f8dd-l98gc in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663879396+08:00 I0127 07:16:11.663627   44050 util.go:206] Pod tke-notify-controller-85cf89b5bb-p9bbx in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663886742+08:00 I0127 07:16:11.663634   44050 util.go:206] Pod kube-proxy-26g7w in namespace kube-system does not Request for GPU resource
2022-01-27T15:16:11.663893465+08:00 I0127 07:16:11.663641   44050 util.go:206] Pod tke-business-api-77ff89ff7-kwcnw in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663900131+08:00 I0127 07:16:11.663649   44050 util.go:206] Pod tke-logagent-api-6d9b6fc4-fbsqk in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663906393+08:00 I0127 07:16:11.663656   44050 util.go:206] Pod tke-business-controller-86879f65df-rz2kv in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663912540+08:00 I0127 07:16:11.663679   44050 util.go:206] Pod tke-notify-api-df68cb499-qtnwz in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663918922+08:00 I0127 07:16:11.663687   44050 util.go:206] Pod tke-registry-api-55b6f778f7-j849x in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663925209+08:00 I0127 07:16:11.663695   44050 util.go:206] Pod tke-registry-controller-5bbb898bc5-r8p26 in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663931196+08:00 I0127 07:16:11.663703   44050 util.go:206] Pod gpu-quota-admission-76cfff49b6-gs8b5 in namespace kube-system does not Request for GPU resource
2022-01-27T15:16:11.663937282+08:00 I0127 07:16:11.663712   44050 util.go:206] Pod tke-auth-controller-5fbbdd9dd9-qj5fx in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.663943414+08:00 I0127 07:16:11.663719   44050 util.go:206] Pod kube-controller-manager-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:16:11.664006182+08:00 I0127 07:16:11.663727   44050 util.go:206] Pod tke-monitor-api-54bc7f44f6-7pl4s in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.664027932+08:00 I0127 07:16:11.663735   44050 util.go:206] Pod kube-apiserver-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:16:11.664038942+08:00 I0127 07:16:11.663742   44050 util.go:206] Pod flannel-pmfcv in namespace kube-system does not Request for GPU resource
2022-01-27T15:16:11.664050282+08:00 I0127 07:16:11.663750   44050 util.go:206] Pod tke-application-controller-6c769cc9cc-8v4zc in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.664059222+08:00 I0127 07:16:11.663758   44050 util.go:206] Pod coredns-6c654f5cc6-wsg7s in namespace kube-system does not Request for GPU resource
2022-01-27T15:16:11.664066719+08:00 I0127 07:16:11.663765   44050 util.go:206] Pod tke-monitor-controller-bb56765d-5vl2x in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.664073649+08:00 I0127 07:16:11.663771   44050 util.go:206] Pod kube-scheduler-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:16:11.664080656+08:00 I0127 07:16:11.663778   44050 util.go:206] Pod etcd-192.168.41.104 in namespace kube-system does not Request for GPU resource
2022-01-27T15:16:11.664087666+08:00 I0127 07:16:11.663785   44050 util.go:206] Pod tke-platform-controller-546fc4fc97-sbp9l in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.664094399+08:00 I0127 07:16:11.663798   44050 util.go:206] Pod tke-logagent-api-6d9b6fc4-7mg2r in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.664104456+08:00 I0127 07:16:11.663806   44050 util.go:206] Pod tke-logagent-controller-c6c8bcbf6-wc7bf in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.664127939+08:00 I0127 07:16:11.663812   44050 util.go:206] Pod tke-application-controller-6c769cc9cc-lbf2s in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.664140716+08:00 I0127 07:16:11.663820   44050 util.go:206] Pod metrics-server-v0.3.6-794ccd69c8-8sqxb in namespace kube-system does not Request for GPU resource
2022-01-27T15:16:11.664150160+08:00 I0127 07:16:11.663827   44050 util.go:206] Pod tke-notify-api-df68cb499-hzc8w in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.664157103+08:00 I0127 07:16:11.663835   44050 util.go:206] Pod tke-auth-api-6fcfdb454d-srxtq in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.664164113+08:00 I0127 07:16:11.663842   44050 util.go:206] Pod tke-business-controller-86879f65df-5wf94 in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.664171170+08:00 I0127 07:16:11.663849   44050 util.go:206] Pod tke-platform-api-b4d45f4bc-4z5ln in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.664178167+08:00 I0127 07:16:11.663857   44050 util.go:206] Pod tke-monitor-controller-bb56765d-78p6v in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.664185153+08:00 I0127 07:16:11.663864   44050 util.go:206] Pod tke-application-api-6db6c5f8dd-tbf2x in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.664192213+08:00 I0127 07:16:11.663871   44050 util.go:206] Pod tke-logagent-controller-c6c8bcbf6-gc6z6 in namespace tke does not Request for GPU resource
2022-01-27T15:16:11.664198790+08:00 I0127 07:16:11.663878   44050 util.go:206] Pod gpu-manager-daemonset-vv7zz in namespace kube-system does not Request for GPU resource
2022-01-27T15:16:11.935739239+08:00 I0127 07:16:11.935632   44050 allocator.go:216] Checking allocation of pods on this node
paragor commented 2 years ago

looks like https://github.com/tkestack/vcuda-controller/blob/master/src/loader.c#L943 cant parse containerd's cgroup

seanchen022 commented 2 years ago

@paragor so gpu-manager cant support containerd?

paragor commented 2 years ago

nvidia hook (nvidia-container-runtime) disabled, right?

i can prepare MR to fix vanilla containerd

seanchen022 commented 2 years ago

@paragor I can create gpu pod by gpu-manager, but when i execute the command nvidia-smi in the gpu pod, i get the following errors: F0627 09:09:43.017690 18 client.go:78] fail to get response from manager, error rpc error: code = Unknown desc = can't find containerd-918d8cef52c37c77df775137619b3ccffcf626cdd0d57d71a82d6113a6df5dcd from docker /tmp/cuda-control/src/register.c:87 rpc client exit with 255 Do you know how to solve this problem?

seanchen022 commented 2 years ago

@guofengtd My cgroup driver is systemd, runtime is containerd and the version of kubernetes is 1.23

paragor commented 2 years ago

1 minutes

paragor commented 2 years ago

@seanchen022 please, try this fix https://github.com/tkestack/gpu-manager/compare/master...paragor:master

i will have gpu cluster only after 4 days for tests :(

DennisYoung96 commented 2 years ago

@seanchen022 i had the same problem like u days ago.
when i change the images/codes of gpu-manager newer, it solved. maybe the branch master, commit: f0669de works. @paragor by the way, have u tried it ur way?