Closed globalmaster closed 3 years ago
I also has this problem
I also has this problem:
[CUDA Bandwidth Test] - Starting...
Running on...
Device 0: GeForce GTX 1050 Ti
Quick Mode
2021-04-02 08:13:16.457533 Gemini E/ attempt 1: Connection refused
2021-04-02 08:13:26.457713 Gemini E/ attempt 2: Connection refused
2021-04-02 08:13:36.457876 Gemini E/ attempt 3: Connection refused
2021-04-02 08:13:46.458070 Gemini E/ attempt 4: Connection refused
2021-04-02 08:13:56.458271 Gemini E/ attempt 5: Connection refused
2021-04-02 08:14:06.458390 Gemini E/ Connection error: Connection refused
Where is this configuration problem?
Hi, could you provide the logs of both kubeshare-device-manager and crashed kubeshare-node-daemon?
kubectl -n kube-system logs kubeshare-device-manager
kubectl -n kube-system logs kubeshare-node-daemon -c config-client
kubectl -n kube-system logs kubeshare-node-daemon -c gemini-scheduler
apiVersion: kubeshare.nthu/v1
kind: SharePod
metadata:
name: pod-test
annotations:
"kubeshare/gpu_request": "0.1" # required if allocating GPU
"kubeshare/gpu_limit": "0.5" # required if allocating GPU
"kubeshare/gpu_mem": "1073741824" # required if allocating GPU # 1Gi, in bytes
"kubeshare/sched_affinity": "red" # optional
"kubeshare/sched_anti-affinity": "green" # optional
"kubeshare/sched_exclusion": "blue" # optional
spec: # PodSpec
containers:
- name: pod-kubeshare-cuda
image: nvidia/cuda:9.0-devel-ubuntu17.04
command: ["sleep"]
args: ["100000"]
volumeMounts:
- mountPath: /home/cuda-sample
name: cuda-sample
volumes:
- name: cuda-sample
hostPath:
path: /home/yb/NVIDIA_CUDA-9.0_Samples
minikube stop
minikube delete --all
sudo systemctl daemon-reload
sudo systemctl restart docker
minikube start --driver=none --apiserver-ips 127.0.0.1 --apiserver-name localhost
kubectl apply -f /home/yb/go/src/k8s-device-plugin/nvidia-device-plugin.yml
kubectl apply -f /home/yb/go/src/KubeShare/crd.yaml
kubectl apply -f /home/yb/go/src/KubeShare/device-manager.yaml
kubectl apply -f /home/yb/go/src/KubeShare/scheduler.yaml
kubectl apply -f /home/yb/go/src/KubeShare/pod_test.yml
the output:
✋ Stopping node "minikube" ...
🛑 1 nodes stopped.
🔄 Uninstalling Kubernetes v1.20.2 using kubeadm ...
🔥 Deleting "minikube" in none ...
💀 Removed all traces of the "minikube" cluster.
🔥 Successfully deleted all profiles
😄 minikube v1.18.1 on Ubuntu 18.04
✨ Using the none driver based on user configuration
👍 Starting control plane node minikube in cluster minikube
🤹 Running on localhost (CPUs=16, Memory=32178MB, Disk=479398MB) ...
ℹ️ OS release is Ubuntu 18.04.5 LTS
🐳 Preparing Kubernetes v1.20.2 on Docker 20.10.5 ...
▪ kubelet.resolv-conf=/run/systemd/resolve/resolv.conf
❗ This bare metal machine is having trouble accessing https://k8s.gcr.io
💡 To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
▪ Generating certificates and keys ...
▪ Booting up control plane ...
▪ Configuring RBAC rules ...
🤹 Configuring local host environment ...
❗ The 'none' driver is designed for experts who need to integrate with an existing VM
💡 Most users should use the newer 'docker' driver instead, which does not require root!
📘 For more information, see: https://minikube.sigs.k8s.io/docs/reference/drivers/none/
❗ kubectl and minikube configuration will be stored in /home/yb
❗ To use kubectl or minikube commands as your own user, you may need to relocate them. For example, to overwrite your own settings, run:
▪ sudo mv /home/yb/.kube /home/yb/.minikube $HOME
▪ sudo chown -R $USER $HOME/.kube $HOME/.minikube
💡 This can also be done automatically by setting the env var CHANGE_MINIKUBE_NONE_USER=true
🔎 Verifying Kubernetes components...
▪ Using image gcr.io/k8s-minikube/storage-provisioner:v4
🌟 Enabled addons: default-storageclass, storage-provisioner
🏄 Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
daemonset.apps/nvidia-device-plugin-daemonset created
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
customresourcedefinition.apiextensions.k8s.io/sharepods.kubeshare.nthu created
serviceaccount/kubeshare-device-manager created
clusterrole.rbac.authorization.k8s.io/kubeshare-device-manager created
clusterrolebinding.rbac.authorization.k8s.io/kubeshare-device-manager created
service/kubeshare-device-manager created
pod/kubeshare-device-manager created
daemonset.apps/kubeshare-node-daemon created
serviceaccount/kubeshare-scheduler created
clusterrole.rbac.authorization.k8s.io/kubeshare-scheduler created
clusterrolebinding.rbac.authorization.k8s.io/kubeshare-scheduler created
pod/kubeshare-scheduler created
sharepod.kubeshare.nthu/pod-test created
Then show the status:
└> kubectl get pods -A
NAMESPACE NAME READY STATUS RESTARTS AGE
default pod-test 1/1 Running 0 52s
kube-system coredns-74ff55c5b-mg6nr 1/1 Running 0 97s
kube-system etcd-yb-server 1/1 Running 0 106s
kube-system kube-apiserver-yb-server 1/1 Running 0 106s
kube-system kube-controller-manager-yb-server 1/1 Running 0 106s
kube-system kube-proxy-tcqrr 1/1 Running 0 98s
kube-system kube-scheduler-yb-server 1/1 Running 0 106s
kube-system kubeshare-device-manager 1/1 Running 0 105s
kube-system kubeshare-node-daemon-ln4wm 2/2 Running 0 96s
kube-system kubeshare-scheduler 1/1 Running 0 104s
kube-system kubeshare-vgpu-yb-server-cnjys 1/1 Running 0 55s
kube-system nvidia-device-plugin-daemonset-dpdnx 1/1 Running 0 97s
kube-system storage-provisioner 1/1 Running 0 112s
└> kubectl get sharepods.kubeshare.nthu -A
NAMESPACE NAME AGE
default pod-test 68s
└> kubectl exec -it pod-test -- bash
root@pod-test:/# home/cuda-sample/1_Utilities/deviceQuery/deviceQuery
home/cuda-sample/1_Utilities/deviceQuery/deviceQuery Starting...
CUDA Device Query (Runtime API) version (CUDART static linking)
Detected 1 CUDA Capable device(s)
Device 0: "GeForce GTX 1050 Ti"
CUDA Driver Version / Runtime Version 11.2 / 9.0
CUDA Capability Major/Minor version number: 6.1
Total amount of global memory: 4032 MBytes (4227858432 bytes)
( 6) Multiprocessors, (128) CUDA Cores/MP: 768 CUDA Cores
GPU Max Clock rate: 1392 MHz (1.39 GHz)
Memory Clock rate: 3504 Mhz
Memory Bus Width: 128-bit
L2 Cache Size: 1048576 bytes
Maximum Texture Dimension Size (x,y,z) 1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
Maximum Layered 1D Texture Size, (num) layers 1D=(32768), 2048 layers
Maximum Layered 2D Texture Size, (num) layers 2D=(32768, 32768), 2048 layers
Total amount of constant memory: 65536 bytes
Total amount of shared memory per block: 49152 bytes
Total number of registers available per block: 65536
Warp size: 32
Maximum number of threads per multiprocessor: 2048
Maximum number of threads per block: 1024
Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
Max dimension size of a grid size (x,y,z): (2147483647, 65535, 65535)
Maximum memory pitch: 2147483647 bytes
Texture alignment: 512 bytes
Concurrent copy and kernel execution: Yes with 2 copy engine(s)
Run time limit on kernels: No
Integrated GPU sharing Host Memory: No
Support host page-locked memory mapping: Yes
Alignment requirement for Surfaces: Yes
Device has ECC support: Disabled
Device supports Unified Addressing (UVA): Yes
Supports Cooperative Kernel Launch: Yes
Supports MultiDevice Co-op Kernel Launch: Yes
Device PCI Domain ID / Bus ID / location ID: 0 / 38 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 11.2, CUDA Runtime Version = 9.0, NumDevs = 1
Result = PASS
root@pod-test:/# home/cuda-sample/1_Utilities/bandwidthTest/bandwidthTest
[CUDA Bandwidth Test] - Starting...
Running on...
Device 0: GeForce GTX 1050 Ti
Quick Mode
2021-04-08 03:46:40.073559 Gemini E/ attempt 1: Connection refused
2021-04-08 03:46:50.073747 Gemini E/ attempt 2: Connection refused
2021-04-08 03:47:00.073916 Gemini E/ attempt 3: Connection refused
2021-04-08 03:47:10.074102 Gemini E/ attempt 4: Connection refused
2021-04-08 03:47:20.074278 Gemini E/ attempt 5: Connection refused
2021-04-08 03:47:30.074520 Gemini E/ Connection error: Connection refused
└> kubectl -n kube-system logs kubeshare-device-manager
W0408 03:41:50.892727 1 client_config.go:543] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0408 03:41:50.907437 1 controller.go:89] Creating event broadcaster
I0408 03:41:50.907524 1 controller.go:106] Setting up event handlers
I0408 03:41:50.907554 1 controller.go:148] Starting SharePod controller
I0408 03:41:50.907559 1 controller.go:151] Waiting for informer caches to sync
I0408 03:41:50.907601 1 reflector.go:150] Starting reflector *v1.Pod (30s) from pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105
I0408 03:41:50.907605 1 reflector.go:150] Starting reflector *v1.SharePod (30s) from pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105
I0408 03:41:50.907621 1 reflector.go:185] Listing and watching *v1.SharePod from pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105
I0408 03:41:50.907616 1 reflector.go:185] Listing and watching *v1.Pod from pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105
I0408 03:41:50.916673 1 controller.go:417] Processing object: storage-provisioner
I0408 03:41:50.916693 1 controller.go:417] Processing object: etcd-yb-server
I0408 03:41:50.916700 1 controller.go:417] Processing object: kube-apiserver-yb-server
I0408 03:41:50.916705 1 controller.go:417] Processing object: kube-proxy-tcqrr
I0408 03:41:50.916712 1 controller.go:417] Processing object: kubeshare-node-daemon-ln4wm
I0408 03:41:50.916717 1 controller.go:417] Processing object: kube-controller-manager-yb-server
I0408 03:41:50.916723 1 controller.go:417] Processing object: kube-scheduler-yb-server
I0408 03:41:50.916729 1 controller.go:417] Processing object: kubeshare-device-manager
I0408 03:41:50.916734 1 controller.go:417] Processing object: kubeshare-scheduler
I0408 03:41:50.916739 1 controller.go:417] Processing object: coredns-74ff55c5b-mg6nr
I0408 03:41:50.916745 1 controller.go:417] Processing object: nvidia-device-plugin-daemonset-dpdnx
I0408 03:41:51.007724 1 shared_informer.go:227] caches populated
I0408 03:41:51.007813 1 controller.go:164] Starting workers
I0408 03:41:51.007824 1 controller.go:170] Started workers
I0408 03:41:51.007882 1 config.go:52] Start listening on 0.0.0.0:9797...
I0408 03:41:51.008066 1 config.go:64] Waiting for clients...
I0408 03:41:51.328898 1 controller.go:417] Processing object: kube-apiserver-yb-server
I0408 03:41:51.723447 1 controller.go:417] Processing object: kube-apiserver-yb-server
I0408 03:41:52.123063 1 controller.go:417] Processing object: kube-proxy-tcqrr
I0408 03:41:52.522667 1 controller.go:417] Processing object: nvidia-device-plugin-daemonset-dpdnx
I0408 03:41:52.922448 1 controller.go:417] Processing object: kubeshare-scheduler
I0408 03:41:53.324234 1 controller.go:417] Processing object: kubeshare-device-manager
I0408 03:41:53.722917 1 controller.go:417] Processing object: coredns-74ff55c5b-mg6nr
I0408 03:41:53.967736 1 controller.go:417] Processing object: storage-provisioner
I0408 03:41:54.124796 1 controller.go:417] Processing object: kubeshare-node-daemon-ln4wm
I0408 03:41:54.725559 1 controller.go:417] Processing object: kubeshare-node-daemon-ln4wm
I0408 03:41:55.121752 1 controller.go:417] Processing object: storage-provisioner
I0408 03:41:55.952227 1 controller.go:417] Processing object: storage-provisioner
I0408 03:41:58.294783 1 controller.go:417] Processing object: coredns-74ff55c5b-mg6nr
I0408 03:42:08.084110 1 config.go:71] Connect to a client, addr:port=172.17.0.1:28224
I0408 03:42:08.093341 1 config.go:110] Receive device list from node: yb-server, devices: GPU-eec6cdf6-8303-ebfa-dc29-17b65addd74f:4032,
I0408 03:42:08.096076 1 config.go:211] Update node yb-server GPU info: GPU-eec6cdf6-8303-ebfa-dc29-17b65addd74f:4032,
I0408 03:42:08.102500 1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:42:20.908569 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:42:20.916805 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:42:23.090453 1 config.go:188] Receive heartbeat from node: yb-server
E0408 03:42:31.979111 1 controller.go:259] SharePod 'default/pod-test' must be scheduled! Spec.NodeName is empty.
I0408 03:42:31.979135 1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:42:31.984405 1 controller.go:315] SharePod default/pod-test is waiting for dummy Pod
I0408 03:42:31.984418 1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:42:31.987315 1 gpupool.go:311] ERICYEH: creating dummy pod: kubeshare-vgpu-yb-server-cnjys
I0408 03:42:31.993568 1 controller.go:417] Processing object: kubeshare-vgpu-yb-server-cnjys
I0408 03:42:32.010455 1 controller.go:417] Processing object: kubeshare-vgpu-yb-server-cnjys
I0408 03:42:34.143877 1 controller.go:417] Processing object: kubeshare-vgpu-yb-server-cnjys
I0408 03:42:34.143893 1 controller.go:437] Start go routine to get UUID from dummy Pod
I0408 03:42:34.155643 1 gpupool.go:396] Dummy Pod kubeshare-vgpu-yb-server-cnjys get device ID: 'GPU-eec6cdf6-8303-ebfa-dc29-17b65addd74f'
I0408 03:42:34.155654 1 gpupool.go:419] After dummy Pod created, PodList Len: 1
I0408 03:42:34.155659 1 gpupool.go:421] Add MtgpuPod back to queue then process: &{default/pod-test %!s(float64=0.1) %!s(float64=0.5) %!s(int64=1073741824) %!s(int=50051)}
I0408 03:42:34.155718 1 config.go:279] Syncing to node 'yb-server' with content: 'GPU-eec6cdf6-8303-ebfa-dc29-17b65addd74f:default/pod-test 0.100000 0.500000 1073741824,:default/pod-test 50051,
'
I0408 03:42:34.155756 1 controller.go:313] SharePod default/pod-test is bound to GPU uuid: GPU-eec6cdf6-8303-ebfa-dc29-17b65addd74f
I0408 03:42:34.174986 1 controller.go:417] Processing object: pod-test
I0408 03:42:34.181423 1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:42:34.181520 1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"549", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:42:34.184929 1 controller.go:417] Processing object: pod-test
E0408 03:42:34.193325 1 controller.go:233] error syncing 'default/pod-test': Operation cannot be fulfilled on sharepods.kubeshare.nthu "pod-test": the object has been modified; please apply your changes to the latest version and try again, requeuing
I0408 03:42:34.204381 1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:42:34.204447 1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"560", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:42:34.214605 1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:42:34.214666 1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"563", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:42:36.304207 1 controller.go:417] Processing object: pod-test
I0408 03:42:36.324153 1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:42:36.324214 1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"563", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:42:36.332764 1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:42:36.332823 1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"571", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:42:38.090625 1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:42:43.313666 1 controller.go:417] Processing object: etcd-yb-server
I0408 03:42:46.314601 1 controller.go:417] Processing object: kube-scheduler-yb-server
I0408 03:42:50.908691 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:42:50.916956 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:42:50.919467 1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:42:50.919533 1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"571", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:42:53.090636 1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:43:04.323042 1 controller.go:417] Processing object: kube-controller-manager-yb-server
I0408 03:43:08.090446 1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:43:20.908859 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:43:20.917130 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:43:20.920323 1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:43:20.920353 1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"571", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:43:23.090708 1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:43:38.090425 1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:43:50.909010 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:43:50.917275 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:43:50.920341 1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:43:50.920408 1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"571", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:43:53.090639 1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:44:08.090477 1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:44:20.909189 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:44:20.917449 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:44:20.929292 1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:44:20.929323 1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"571", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:44:23.090550 1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:44:38.090623 1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:44:50.909306 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:44:50.917590 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:44:50.934248 1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:44:50.934314 1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"571", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:44:53.090597 1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:45:08.090599 1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:45:20.909427 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:45:20.917712 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:45:20.929110 1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:45:20.929175 1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"571", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:45:23.090470 1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:45:38.090491 1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:45:50.909584 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:45:50.917860 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:45:50.929843 1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:45:50.929911 1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"571", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:45:53.090460 1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:46:08.090672 1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:46:20.909718 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:46:20.918038 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:46:20.930071 1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:46:20.930134 1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"571", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:46:23.090494 1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:46:38.090635 1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:46:50.909853 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:46:50.918167 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:46:50.934214 1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:46:50.934301 1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"571", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:46:53.090539 1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:47:08.090507 1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:47:20.909988 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:47:20.918288 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:47:20.933855 1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:47:20.933924 1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"571", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:47:23.090483 1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:47:38.090649 1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:47:50.910146 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:47:50.918437 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:47:50.930201 1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:47:50.930264 1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"571", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:47:53.090673 1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:48:08.090494 1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:48:20.910888 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:48:20.918608 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:48:20.938756 1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:48:20.938802 1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"571", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:48:23.090682 1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:48:38.090680 1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:48:50.911267 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:48:50.918750 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:48:50.935611 1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:48:50.935678 1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"571", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:48:53.090532 1 config.go:188] Receive heartbeat from node: yb-server
└> kubectl -n kube-system logs kubeshare-node-daemon-ln4wm -c config-client
2021/04/08 03:41:53 Loading NVML
I0408 03:42:08.084062 1 config-client.go:48] Connect successed.
I0408 03:42:08.090180 1 config-client.go:80] Registering nvidia device to server in registerDevices(), msg: GPU-eec6cdf6-8303-ebfa-dc29-17b65addd74f:4032,
I0408 03:42:08.090252 1 config-client.go:129] Send heartbeat: 2021-04-08 03:42:08.090236735 +0000 UTC m=+15.027826250
I0408 03:42:23.090328 1 config-client.go:133] Send heartbeat: 2021-04-08 03:42:23.090308081 +0000 UTC m=+30.027897646
I0408 03:42:34.155786 1 config-client.go:96] Receive request: GPU-eec6cdf6-8303-ebfa-dc29-17b65addd74f:default/pod-test 0.100000 0.500000 1073741824,:default/pod-test 50051,
I0408 03:42:38.090528 1 config-client.go:133] Send heartbeat: 2021-04-08 03:42:38.090505201 +0000 UTC m=+45.028094806
I0408 03:42:53.090526 1 config-client.go:133] Send heartbeat: 2021-04-08 03:42:53.090507023 +0000 UTC m=+60.028096578
I0408 03:43:08.090350 1 config-client.go:133] Send heartbeat: 2021-04-08 03:43:08.0903339 +0000 UTC m=+75.027923455
I0408 03:43:23.090582 1 config-client.go:133] Send heartbeat: 2021-04-08 03:43:23.090564583 +0000 UTC m=+90.028154148
I0408 03:43:38.090329 1 config-client.go:133] Send heartbeat: 2021-04-08 03:43:38.090312109 +0000 UTC m=+105.027901674
I0408 03:43:53.090514 1 config-client.go:133] Send heartbeat: 2021-04-08 03:43:53.090496557 +0000 UTC m=+120.028086052
I0408 03:44:08.090355 1 config-client.go:133] Send heartbeat: 2021-04-08 03:44:08.090335203 +0000 UTC m=+135.027924778
I0408 03:44:23.090330 1 config-client.go:133] Send heartbeat: 2021-04-08 03:44:23.090313033 +0000 UTC m=+150.027902608
I0408 03:44:38.090515 1 config-client.go:133] Send heartbeat: 2021-04-08 03:44:38.090497771 +0000 UTC m=+165.028087266
I0408 03:44:53.090497 1 config-client.go:133] Send heartbeat: 2021-04-08 03:44:53.090475901 +0000 UTC m=+180.028065486
I0408 03:45:08.090348 1 config-client.go:133] Send heartbeat: 2021-04-08 03:45:08.090329706 +0000 UTC m=+195.027919251
I0408 03:45:23.090355 1 config-client.go:133] Send heartbeat: 2021-04-08 03:45:23.090334959 +0000 UTC m=+210.027924514
I0408 03:45:38.090380 1 config-client.go:133] Send heartbeat: 2021-04-08 03:45:38.090362525 +0000 UTC m=+225.027952020
I0408 03:45:53.090370 1 config-client.go:133] Send heartbeat: 2021-04-08 03:45:53.09035077 +0000 UTC m=+240.027940365
I0408 03:46:08.090559 1 config-client.go:133] Send heartbeat: 2021-04-08 03:46:08.090539341 +0000 UTC m=+255.028128926
I0408 03:46:23.090362 1 config-client.go:133] Send heartbeat: 2021-04-08 03:46:23.090344426 +0000 UTC m=+270.027934021
I0408 03:46:38.090542 1 config-client.go:133] Send heartbeat: 2021-04-08 03:46:38.090524411 +0000 UTC m=+285.028113986
I0408 03:46:53.090349 1 config-client.go:133] Send heartbeat: 2021-04-08 03:46:53.090330706 +0000 UTC m=+300.027920271
I0408 03:47:08.090389 1 config-client.go:133] Send heartbeat: 2021-04-08 03:47:08.090372353 +0000 UTC m=+315.027961938
I0408 03:47:23.090383 1 config-client.go:133] Send heartbeat: 2021-04-08 03:47:23.090365688 +0000 UTC m=+330.027955283
I0408 03:47:38.090545 1 config-client.go:133] Send heartbeat: 2021-04-08 03:47:38.090526957 +0000 UTC m=+345.028116532
I0408 03:47:53.090541 1 config-client.go:133] Send heartbeat: 2021-04-08 03:47:53.090523383 +0000 UTC m=+360.028112978
I0408 03:48:08.090376 1 config-client.go:133] Send heartbeat: 2021-04-08 03:48:08.090352219 +0000 UTC m=+375.027941814
I0408 03:48:23.090566 1 config-client.go:133] Send heartbeat: 2021-04-08 03:48:23.090547824 +0000 UTC m=+390.028137389
I0408 03:48:38.090540 1 config-client.go:133] Send heartbeat: 2021-04-08 03:48:38.090520345 +0000 UTC m=+405.028109930
I0408 03:48:53.090386 1 config-client.go:133] Send heartbeat: 2021-04-08 03:48:53.090364341 +0000 UTC m=+420.027953936
I0408 03:49:08.090386 1 config-client.go:133] Send heartbeat: 2021-04-08 03:49:08.090367371 +0000 UTC m=+435.027956916
I0408 03:49:23.090357 1 config-client.go:133] Send heartbeat: 2021-04-08 03:49:23.090336605 +0000 UTC m=+450.027926100
I0408 03:49:38.090680 1 config-client.go:133] Send heartbeat: 2021-04-08 03:49:38.090661802 +0000 UTC m=+465.028251347
I0408 03:49:53.090381 1 config-client.go:133] Send heartbeat: 2021-04-08 03:49:53.090363113 +0000 UTC m=+480.027952688
I0408 03:50:08.090527 1 config-client.go:133] Send heartbeat: 2021-04-08 03:50:08.090507825 +0000 UTC m=+495.028097380
I0408 03:50:23.090518 1 config-client.go:133] Send heartbeat: 2021-04-08 03:50:23.090499894 +0000 UTC m=+510.028089489
I0408 03:50:38.090559 1 config-client.go:133] Send heartbeat: 2021-04-08 03:50:38.09054086 +0000 UTC m=+525.028130425
I0408 03:50:53.090338 1 config-client.go:133] Send heartbeat: 2021-04-08 03:50:53.090320467 +0000 UTC m=+540.027909962
I0408 03:51:08.090367 1 config-client.go:133] Send heartbeat: 2021-04-08 03:51:08.090349431 +0000 UTC m=+555.027938926
I0408 03:51:23.090369 1 config-client.go:133] Send heartbeat: 2021-04-08 03:51:23.09035098 +0000 UTC m=+570.027940585
└> kubectl -n kube-system logs kubeshare-node-daemon-ln4wm -c gemini-scheduler
/usr/bin/nvidia-smi
[launcher] scheduler started on 0.0.0.0:49901
2021-04-08 03:41:53.571041 Gemini I/ There are 0 clients in the system...
2021-04-08 03:41:53.571168 Gemini I/ Monitor thread created.
2021-04-08 03:41:53.571180 Gemini I/ Waiting for incoming connection
2021-04-08 03:41:53.571196 Gemini I/ Watching '/kubeshare/scheduler/config'.
2021-04-08 03:42:34.165974 Gemini I/ File GPU-eec6cdf6-8303-ebfa-dc29-17b65addd74f modified with watch descriptor 1.
2021-04-08 03:42:34.165993 Gemini I/ Update containers' settings...
2021-04-08 03:42:34.166041 Gemini I/ There are 1 clients in the system...
2021-04-08 03:42:34.166080 Gemini I/ default/pod-test request: 0.10, limit: 0.50, memory limit: 1073741824 bytes
[launcher] pod manager id 'default/pod-test 50051' port '50051' start running
2021-04-08 03:42:34.169123 Gemini I/ Pod server port = 50051.
2021-04-08 03:42:34.169140 Gemini I/ scheduler 127.0.0.1:49901
2021-04-08 03:42:34.169230 Gemini I/ Received an incoming connection.
2021-04-08 03:42:34.169313 Gemini I/ GPU memory limit: 1073741824 bytes.
└> kubectl -n kube-system logs kubeshare-scheduler
W0408 03:41:50.694588 1 client_config.go:543] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0408 03:41:50.768486 1 controller.go:83] Creating event broadcaster
I0408 03:41:50.768651 1 controller.go:104] Setting up event handlers
I0408 03:41:50.768704 1 controller.go:126] Starting Foo controller
I0408 03:41:50.768711 1 controller.go:128] Waiting for informer caches to sync
I0408 03:41:50.768898 1 reflector.go:150] Starting reflector *v1.Node (30s) from pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105
I0408 03:41:50.768901 1 reflector.go:150] Starting reflector *v1.Pod (30s) from pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105
I0408 03:41:50.768918 1 reflector.go:185] Listing and watching *v1.Node from pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105
I0408 03:41:50.768922 1 reflector.go:185] Listing and watching *v1.Pod from pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105
I0408 03:41:50.768934 1 reflector.go:150] Starting reflector *v1.SharePod (30s) from pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105
I0408 03:41:50.768957 1 reflector.go:185] Listing and watching *v1.SharePod from pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105
I0408 03:41:50.868874 1 shared_informer.go:227] caches populated
I0408 03:41:50.868893 1 controller.go:133] Starting workers
I0408 03:41:50.868913 1 controller.go:142] Started workers
I0408 03:42:20.770545 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:42:20.772462 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:42:20.779372 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:42:31.975969 1 controller.go:268] SharePod 'default/pod-test' had been scheduled to node 'yb-server' GPUID 'cnjys'.
I0408 03:42:31.983826 1 controller.go:180] Successfully synced 'default/pod-test'
I0408 03:42:31.983937 1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"548", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod scheduled successfully
I0408 03:42:50.770701 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:42:50.772581 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:42:50.779521 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:43:20.770828 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:43:20.772707 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:43:20.779664 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:43:50.770961 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:43:50.772828 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:43:50.779776 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:44:20.771112 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:44:20.772965 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:44:20.779935 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:44:50.771253 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:44:50.773071 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:44:50.780054 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:45:20.771372 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:45:20.773176 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:45:20.780174 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:45:50.771515 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:45:50.773295 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:45:50.780327 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:46:20.771654 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:46:20.773393 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:46:20.780520 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:46:50.771766 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:46:50.773503 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:46:50.780665 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:47:20.772088 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:47:20.773646 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:47:20.780792 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:47:50.772225 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:47:50.773768 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:47:50.781103 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:48:20.772397 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:48:20.773920 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:48:20.781274 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:48:50.772560 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:48:50.774067 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:48:50.781439 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:49:20.772814 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:49:20.774189 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:49:20.781560 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:49:50.772985 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:49:50.774312 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:49:50.781705 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:49:51.771081 1 reflector.go:418] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Watch close - *v1.SharePod total 13 items received
I0408 03:50:09.780213 1 reflector.go:418] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Watch close - *v1.Pod total 31 items received
I0408 03:50:20.773150 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:50:20.774433 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:50:20.781858 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:50:50.773287 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:50:50.774542 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:50:50.781971 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:51:20.773429 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:51:20.774667 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:51:20.782114 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:51:32.772933 1 reflector.go:418] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Watch close - *v1.Node total 13 items received
I0408 03:51:50.773582 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:51:50.774793 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:51:50.782268 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:52:20.773729 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:52:20.774890 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:52:20.782394 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:52:50.773851 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:52:50.775010 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:52:50.782578 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:53:20.773983 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:53:20.775108 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:53:20.782701 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:53:50.774113 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:53:50.775200 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:53:50.782819 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:54:20.774260 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:54:20.775329 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:54:20.783094 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
no logs
I have fixed this problem.I changed the environment variable "POD_MANAGER_IP". The default value of "POD_MANAGER_IP" is wrong.I run command,"export POD_MANAGER_IP = $(head -n 1 /kubeshare/library/schedulerIP.txt)", in container to change the environment variable.
I have fixed this problem.I changed the environment variable "POD_MANAGER_IP". The default value of "POD_MANAGER_IP" is wrong.I run command,"export POD_MANAGER_IP = $(head -n 1 /kubeshare/library/schedulerIP.txt)", in container to change the environment variable.
Yes, it turned out to be this place. I also solved this problem, thank you!!!
Hi,I run:
kubectl logs pod3 -f
get this error:
Can you help me? @ncy9371