search

NTHU-LSALAB / KubeShare

Share GPU between Pods in Kubernetes
Apache License 2.0
193 stars 42 forks source link

Gemini E/ attempt 1: Connection refused #9

Closed globalmaster closed 3 years ago

globalmaster commented 4 years ago

Hi，I run： kubectl logs pod3 -f

get this error:

2020-06-22 09:15:16.220809 Gemini E/ attempt 1: Connection refused
2020-06-22 09:15:26.221053 Gemini E/ attempt 2: Connection refused
2020-06-22 09:15:36.221295 Gemini E/ attempt 3: Connection refused
2020-06-22 09:15:46.221547 Gemini E/ attempt 4: Connection refused
2020-06-22 09:15:56.221820 Gemini E/ attempt 5: Connection refused
2020-06-22 09:16:06.221937 Gemini E/ Connection error: Connection refused

Can you help me? @ncy9371

SirZen97 commented 3 years ago

I also has this problem

yanghaku commented 3 years ago

I also has this problem:

[CUDA Bandwidth Test] - Starting...
Running on...

 Device 0: GeForce GTX 1050 Ti
 Quick Mode

2021-04-02 08:13:16.457533 Gemini E/ attempt 1: Connection refused
2021-04-02 08:13:26.457713 Gemini E/ attempt 2: Connection refused
2021-04-02 08:13:36.457876 Gemini E/ attempt 3: Connection refused
2021-04-02 08:13:46.458070 Gemini E/ attempt 4: Connection refused
2021-04-02 08:13:56.458271 Gemini E/ attempt 5: Connection refused
2021-04-02 08:14:06.458390 Gemini E/ Connection error: Connection refused

Where is this configuration problem?

StarCoral commented 3 years ago

Hi, could you provide the logs of both kubeshare-device-manager and crashed kubeshare-node-daemon?

kubectl -n kube-system logs kubeshare-device-manager
kubectl -n kube-system logs kubeshare-node-daemon -c config-client
kubectl -n kube-system logs kubeshare-node-daemon -c gemini-scheduler
yanghaku commented 3 years ago

Hardware:

Environment:

Details

shared pod file:

apiVersion: kubeshare.nthu/v1
kind: SharePod
metadata:
  name: pod-test
  annotations:
    "kubeshare/gpu_request": "0.1" # required if allocating GPU
    "kubeshare/gpu_limit": "0.5" # required if allocating GPU
    "kubeshare/gpu_mem": "1073741824" # required if allocating GPU # 1Gi, in bytes
    "kubeshare/sched_affinity": "red" # optional
    "kubeshare/sched_anti-affinity": "green" # optional
    "kubeshare/sched_exclusion": "blue" # optional
spec: # PodSpec
  containers:
  - name: pod-kubeshare-cuda
    image: nvidia/cuda:9.0-devel-ubuntu17.04
    command: ["sleep"]
    args: ["100000"]
    volumeMounts:
    - mountPath: /home/cuda-sample
      name: cuda-sample
  volumes:
  - name: cuda-sample
    hostPath:
      path: /home/yb/NVIDIA_CUDA-9.0_Samples

start script:

minikube stop
minikube delete --all

sudo systemctl daemon-reload
sudo systemctl restart docker

minikube start --driver=none --apiserver-ips 127.0.0.1 --apiserver-name localhost

kubectl apply -f /home/yb/go/src/k8s-device-plugin/nvidia-device-plugin.yml

kubectl apply -f /home/yb/go/src/KubeShare/crd.yaml
kubectl apply -f /home/yb/go/src/KubeShare/device-manager.yaml
kubectl apply -f /home/yb/go/src/KubeShare/scheduler.yaml

kubectl apply -f /home/yb/go/src/KubeShare/pod_test.yml

the output:

✋  Stopping node "minikube"  ...
🛑  1 nodes stopped.
🔄  Uninstalling Kubernetes v1.20.2 using kubeadm ...
🔥  Deleting "minikube" in none ...
💀  Removed all traces of the "minikube" cluster.
🔥  Successfully deleted all profiles
😄  minikube v1.18.1 on Ubuntu 18.04
✨  Using the none driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🤹  Running on localhost (CPUs=16, Memory=32178MB, Disk=479398MB) ...
ℹ️  OS release is Ubuntu 18.04.5 LTS
🐳  Preparing Kubernetes v1.20.2 on Docker 20.10.5 ...
    ▪ kubelet.resolv-conf=/run/systemd/resolve/resolv.conf
❗  This bare metal machine is having trouble accessing https://k8s.gcr.io
💡  To pull new external images, you may need to configure a proxy: https://minikube.sigs.k8s.io/docs/reference/networking/proxy/
    ▪ Generating certificates and keys ...
    ▪ Booting up control plane ...
    ▪ Configuring RBAC rules ...
🤹  Configuring local host environment ...

❗  The 'none' driver is designed for experts who need to integrate with an existing VM
💡  Most users should use the newer 'docker' driver instead, which does not require root!
📘  For more information, see: https://minikube.sigs.k8s.io/docs/reference/drivers/none/

❗  kubectl and minikube configuration will be stored in /home/yb
❗  To use kubectl or minikube commands as your own user, you may need to relocate them. For example, to overwrite your own settings, run:

    ▪ sudo mv /home/yb/.kube /home/yb/.minikube $HOME
    ▪ sudo chown -R $USER $HOME/.kube $HOME/.minikube

💡  This can also be done automatically by setting the env var CHANGE_MINIKUBE_NONE_USER=true
🔎  Verifying Kubernetes components...
    ▪ Using image gcr.io/k8s-minikube/storage-provisioner:v4
🌟  Enabled addons: default-storageclass, storage-provisioner
🏄  Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
daemonset.apps/nvidia-device-plugin-daemonset created
Warning: apiextensions.k8s.io/v1beta1 CustomResourceDefinition is deprecated in v1.16+, unavailable in v1.22+; use apiextensions.k8s.io/v1 CustomResourceDefinition
customresourcedefinition.apiextensions.k8s.io/sharepods.kubeshare.nthu created
serviceaccount/kubeshare-device-manager created
clusterrole.rbac.authorization.k8s.io/kubeshare-device-manager created
clusterrolebinding.rbac.authorization.k8s.io/kubeshare-device-manager created
service/kubeshare-device-manager created
pod/kubeshare-device-manager created
daemonset.apps/kubeshare-node-daemon created
serviceaccount/kubeshare-scheduler created
clusterrole.rbac.authorization.k8s.io/kubeshare-scheduler created
clusterrolebinding.rbac.authorization.k8s.io/kubeshare-scheduler created
pod/kubeshare-scheduler created
sharepod.kubeshare.nthu/pod-test created

Then show the status:

└> kubectl get pods -A           

NAMESPACE     NAME                                   READY   STATUS    RESTARTS   AGE
default       pod-test                               1/1     Running   0          52s
kube-system   coredns-74ff55c5b-mg6nr                1/1     Running   0          97s
kube-system   etcd-yb-server                         1/1     Running   0          106s
kube-system   kube-apiserver-yb-server               1/1     Running   0          106s
kube-system   kube-controller-manager-yb-server      1/1     Running   0          106s
kube-system   kube-proxy-tcqrr                       1/1     Running   0          98s
kube-system   kube-scheduler-yb-server               1/1     Running   0          106s
kube-system   kubeshare-device-manager               1/1     Running   0          105s
kube-system   kubeshare-node-daemon-ln4wm            2/2     Running   0          96s
kube-system   kubeshare-scheduler                    1/1     Running   0          104s
kube-system   kubeshare-vgpu-yb-server-cnjys         1/1     Running   0          55s
kube-system   nvidia-device-plugin-daemonset-dpdnx   1/1     Running   0          97s
kube-system   storage-provisioner                    1/1     Running   0          112s

└> kubectl get sharepods.kubeshare.nthu -A

NAMESPACE   NAME       AGE
default     pod-test   68s

run cuda sample in the container

deviceQuery:
└> kubectl exec -it pod-test -- bash          

root@pod-test:/# home/cuda-sample/1_Utilities/deviceQuery/deviceQuery 
home/cuda-sample/1_Utilities/deviceQuery/deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "GeForce GTX 1050 Ti"
  CUDA Driver Version / Runtime Version          11.2 / 9.0
  CUDA Capability Major/Minor version number:    6.1
  Total amount of global memory:                 4032 MBytes (4227858432 bytes)
  ( 6) Multiprocessors, (128) CUDA Cores/MP:     768 CUDA Cores
  GPU Max Clock rate:                            1392 MHz (1.39 GHz)
  Memory Clock rate:                             3504 Mhz
  Memory Bus Width:                              128-bit
  L2 Cache Size:                                 1048576 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 2 copy engine(s)
  Run time limit on kernels:                     No
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  Device supports Unified Addressing (UVA):      Yes
  Supports Cooperative Kernel Launch:            Yes
  Supports MultiDevice Co-op Kernel Launch:      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 38 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 11.2, CUDA Runtime Version = 9.0, NumDevs = 1
Result = PASS
bandwidthTest
root@pod-test:/# home/cuda-sample/1_Utilities/bandwidthTest/bandwidthTest 
[CUDA Bandwidth Test] - Starting...
Running on...

 Device 0: GeForce GTX 1050 Ti
 Quick Mode

2021-04-08 03:46:40.073559 Gemini E/ attempt 1: Connection refused
2021-04-08 03:46:50.073747 Gemini E/ attempt 2: Connection refused
2021-04-08 03:47:00.073916 Gemini E/ attempt 3: Connection refused
2021-04-08 03:47:10.074102 Gemini E/ attempt 4: Connection refused
2021-04-08 03:47:20.074278 Gemini E/ attempt 5: Connection refused
2021-04-08 03:47:30.074520 Gemini E/ Connection error: Connection refused

logs

kubectl device manager


└> kubectl -n kube-system logs kubeshare-device-manager
W0408 03:41:50.892727       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0408 03:41:50.907437       1 controller.go:89] Creating event broadcaster
I0408 03:41:50.907524       1 controller.go:106] Setting up event handlers
I0408 03:41:50.907554       1 controller.go:148] Starting SharePod controller
I0408 03:41:50.907559       1 controller.go:151] Waiting for informer caches to sync
I0408 03:41:50.907601       1 reflector.go:150] Starting reflector *v1.Pod (30s) from pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105
I0408 03:41:50.907605       1 reflector.go:150] Starting reflector *v1.SharePod (30s) from pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105
I0408 03:41:50.907621       1 reflector.go:185] Listing and watching *v1.SharePod from pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105
I0408 03:41:50.907616       1 reflector.go:185] Listing and watching *v1.Pod from pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105
I0408 03:41:50.916673       1 controller.go:417] Processing object: storage-provisioner
I0408 03:41:50.916693       1 controller.go:417] Processing object: etcd-yb-server
I0408 03:41:50.916700       1 controller.go:417] Processing object: kube-apiserver-yb-server
I0408 03:41:50.916705       1 controller.go:417] Processing object: kube-proxy-tcqrr
I0408 03:41:50.916712       1 controller.go:417] Processing object: kubeshare-node-daemon-ln4wm
I0408 03:41:50.916717       1 controller.go:417] Processing object: kube-controller-manager-yb-server
I0408 03:41:50.916723       1 controller.go:417] Processing object: kube-scheduler-yb-server
I0408 03:41:50.916729       1 controller.go:417] Processing object: kubeshare-device-manager
I0408 03:41:50.916734       1 controller.go:417] Processing object: kubeshare-scheduler
I0408 03:41:50.916739       1 controller.go:417] Processing object: coredns-74ff55c5b-mg6nr
I0408 03:41:50.916745       1 controller.go:417] Processing object: nvidia-device-plugin-daemonset-dpdnx
I0408 03:41:51.007724       1 shared_informer.go:227] caches populated
I0408 03:41:51.007813       1 controller.go:164] Starting workers
I0408 03:41:51.007824       1 controller.go:170] Started workers
I0408 03:41:51.007882       1 config.go:52] Start listening on 0.0.0.0:9797...
I0408 03:41:51.008066       1 config.go:64] Waiting for clients...
I0408 03:41:51.328898       1 controller.go:417] Processing object: kube-apiserver-yb-server
I0408 03:41:51.723447       1 controller.go:417] Processing object: kube-apiserver-yb-server
I0408 03:41:52.123063       1 controller.go:417] Processing object: kube-proxy-tcqrr
I0408 03:41:52.522667       1 controller.go:417] Processing object: nvidia-device-plugin-daemonset-dpdnx
I0408 03:41:52.922448       1 controller.go:417] Processing object: kubeshare-scheduler
I0408 03:41:53.324234       1 controller.go:417] Processing object: kubeshare-device-manager
I0408 03:41:53.722917       1 controller.go:417] Processing object: coredns-74ff55c5b-mg6nr
I0408 03:41:53.967736       1 controller.go:417] Processing object: storage-provisioner
I0408 03:41:54.124796       1 controller.go:417] Processing object: kubeshare-node-daemon-ln4wm
I0408 03:41:54.725559       1 controller.go:417] Processing object: kubeshare-node-daemon-ln4wm
I0408 03:41:55.121752       1 controller.go:417] Processing object: storage-provisioner
I0408 03:41:55.952227       1 controller.go:417] Processing object: storage-provisioner
I0408 03:41:58.294783       1 controller.go:417] Processing object: coredns-74ff55c5b-mg6nr
I0408 03:42:08.084110       1 config.go:71] Connect to a client, addr:port=172.17.0.1:28224
I0408 03:42:08.093341       1 config.go:110] Receive device list from node: yb-server, devices: GPU-eec6cdf6-8303-ebfa-dc29-17b65addd74f:4032,
I0408 03:42:08.096076       1 config.go:211] Update node yb-server GPU info: GPU-eec6cdf6-8303-ebfa-dc29-17b65addd74f:4032,
I0408 03:42:08.102500       1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:42:20.908569       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:42:20.916805       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:42:23.090453       1 config.go:188] Receive heartbeat from node: yb-server
E0408 03:42:31.979111       1 controller.go:259] SharePod 'default/pod-test' must be scheduled! Spec.NodeName is empty.
I0408 03:42:31.979135       1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:42:31.984405       1 controller.go:315] SharePod default/pod-test is waiting for dummy Pod
I0408 03:42:31.984418       1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:42:31.987315       1 gpupool.go:311] ERICYEH: creating dummy pod: kubeshare-vgpu-yb-server-cnjys
I0408 03:42:31.993568       1 controller.go:417] Processing object: kubeshare-vgpu-yb-server-cnjys
I0408 03:42:32.010455       1 controller.go:417] Processing object: kubeshare-vgpu-yb-server-cnjys
I0408 03:42:34.143877       1 controller.go:417] Processing object: kubeshare-vgpu-yb-server-cnjys
I0408 03:42:34.143893       1 controller.go:437] Start go routine to get UUID from dummy Pod
I0408 03:42:34.155643       1 gpupool.go:396] Dummy Pod kubeshare-vgpu-yb-server-cnjys get device ID: 'GPU-eec6cdf6-8303-ebfa-dc29-17b65addd74f'
I0408 03:42:34.155654       1 gpupool.go:419] After dummy Pod created, PodList Len: 1
I0408 03:42:34.155659       1 gpupool.go:421] Add MtgpuPod back to queue then process: &{default/pod-test %!s(float64=0.1) %!s(float64=0.5) %!s(int64=1073741824) %!s(int=50051)}
I0408 03:42:34.155718       1 config.go:279] Syncing to node 'yb-server' with content: 'GPU-eec6cdf6-8303-ebfa-dc29-17b65addd74f:default/pod-test 0.100000 0.500000 1073741824,:default/pod-test 50051,
'
I0408 03:42:34.155756       1 controller.go:313] SharePod default/pod-test is bound to GPU uuid: GPU-eec6cdf6-8303-ebfa-dc29-17b65addd74f
I0408 03:42:34.174986       1 controller.go:417] Processing object: pod-test
I0408 03:42:34.181423       1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:42:34.181520       1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"549", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:42:34.184929       1 controller.go:417] Processing object: pod-test
E0408 03:42:34.193325       1 controller.go:233] error syncing 'default/pod-test': Operation cannot be fulfilled on sharepods.kubeshare.nthu "pod-test": the object has been modified; please apply your changes to the latest version and try again, requeuing
I0408 03:42:34.204381       1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:42:34.204447       1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"560", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:42:34.214605       1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:42:34.214666       1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"563", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:42:36.304207       1 controller.go:417] Processing object: pod-test
I0408 03:42:36.324153       1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:42:36.324214       1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"563", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:42:36.332764       1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:42:36.332823       1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"571", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:42:38.090625       1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:42:43.313666       1 controller.go:417] Processing object: etcd-yb-server
I0408 03:42:46.314601       1 controller.go:417] Processing object: kube-scheduler-yb-server
I0408 03:42:50.908691       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:42:50.916956       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:42:50.919467       1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:42:50.919533       1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"571", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:42:53.090636       1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:43:04.323042       1 controller.go:417] Processing object: kube-controller-manager-yb-server
I0408 03:43:08.090446       1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:43:20.908859       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:43:20.917130       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:43:20.920323       1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:43:20.920353       1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"571", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:43:23.090708       1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:43:38.090425       1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:43:50.909010       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:43:50.917275       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:43:50.920341       1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:43:50.920408       1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"571", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:43:53.090639       1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:44:08.090477       1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:44:20.909189       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:44:20.917449       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:44:20.929292       1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:44:20.929323       1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"571", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:44:23.090550       1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:44:38.090623       1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:44:50.909306       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:44:50.917590       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:44:50.934248       1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:44:50.934314       1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"571", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:44:53.090597       1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:45:08.090599       1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:45:20.909427       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:45:20.917712       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:45:20.929110       1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:45:20.929175       1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"571", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:45:23.090470       1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:45:38.090491       1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:45:50.909584       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:45:50.917860       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:45:50.929843       1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:45:50.929911       1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"571", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:45:53.090460       1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:46:08.090672       1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:46:20.909718       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:46:20.918038       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:46:20.930071       1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:46:20.930134       1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"571", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:46:23.090494       1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:46:38.090635       1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:46:50.909853       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:46:50.918167       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:46:50.934214       1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:46:50.934301       1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"571", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:46:53.090539       1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:47:08.090507       1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:47:20.909988       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:47:20.918288       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:47:20.933855       1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:47:20.933924       1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"571", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:47:23.090483       1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:47:38.090649       1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:47:50.910146       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:47:50.918437       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:47:50.930201       1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:47:50.930264       1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"571", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:47:53.090673       1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:48:08.090494       1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:48:20.910888       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:48:20.918608       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:48:20.938756       1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:48:20.938802       1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"571", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:48:23.090682       1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:48:38.090680       1 config.go:188] Receive heartbeat from node: yb-server
I0408 03:48:50.911267       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:48:50.918750       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:48:50.935611       1 controller.go:228] Successfully synced 'default/pod-test'
I0408 03:48:50.935678       1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"571", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod synced successfully
I0408 03:48:53.090532       1 config.go:188] Receive heartbeat from node: yb-server

kubeshare-node-daemon / config-client

└> kubectl -n kube-system logs kubeshare-node-daemon-ln4wm -c config-client 

2021/04/08 03:41:53 Loading NVML
I0408 03:42:08.084062       1 config-client.go:48] Connect successed.
I0408 03:42:08.090180       1 config-client.go:80] Registering nvidia device to server in registerDevices(), msg: GPU-eec6cdf6-8303-ebfa-dc29-17b65addd74f:4032,
I0408 03:42:08.090252       1 config-client.go:129] Send heartbeat: 2021-04-08 03:42:08.090236735 +0000 UTC m=+15.027826250
I0408 03:42:23.090328       1 config-client.go:133] Send heartbeat: 2021-04-08 03:42:23.090308081 +0000 UTC m=+30.027897646
I0408 03:42:34.155786       1 config-client.go:96] Receive request: GPU-eec6cdf6-8303-ebfa-dc29-17b65addd74f:default/pod-test 0.100000 0.500000 1073741824,:default/pod-test 50051,
I0408 03:42:38.090528       1 config-client.go:133] Send heartbeat: 2021-04-08 03:42:38.090505201 +0000 UTC m=+45.028094806
I0408 03:42:53.090526       1 config-client.go:133] Send heartbeat: 2021-04-08 03:42:53.090507023 +0000 UTC m=+60.028096578
I0408 03:43:08.090350       1 config-client.go:133] Send heartbeat: 2021-04-08 03:43:08.0903339 +0000 UTC m=+75.027923455
I0408 03:43:23.090582       1 config-client.go:133] Send heartbeat: 2021-04-08 03:43:23.090564583 +0000 UTC m=+90.028154148
I0408 03:43:38.090329       1 config-client.go:133] Send heartbeat: 2021-04-08 03:43:38.090312109 +0000 UTC m=+105.027901674
I0408 03:43:53.090514       1 config-client.go:133] Send heartbeat: 2021-04-08 03:43:53.090496557 +0000 UTC m=+120.028086052
I0408 03:44:08.090355       1 config-client.go:133] Send heartbeat: 2021-04-08 03:44:08.090335203 +0000 UTC m=+135.027924778
I0408 03:44:23.090330       1 config-client.go:133] Send heartbeat: 2021-04-08 03:44:23.090313033 +0000 UTC m=+150.027902608
I0408 03:44:38.090515       1 config-client.go:133] Send heartbeat: 2021-04-08 03:44:38.090497771 +0000 UTC m=+165.028087266
I0408 03:44:53.090497       1 config-client.go:133] Send heartbeat: 2021-04-08 03:44:53.090475901 +0000 UTC m=+180.028065486
I0408 03:45:08.090348       1 config-client.go:133] Send heartbeat: 2021-04-08 03:45:08.090329706 +0000 UTC m=+195.027919251
I0408 03:45:23.090355       1 config-client.go:133] Send heartbeat: 2021-04-08 03:45:23.090334959 +0000 UTC m=+210.027924514
I0408 03:45:38.090380       1 config-client.go:133] Send heartbeat: 2021-04-08 03:45:38.090362525 +0000 UTC m=+225.027952020
I0408 03:45:53.090370       1 config-client.go:133] Send heartbeat: 2021-04-08 03:45:53.09035077 +0000 UTC m=+240.027940365
I0408 03:46:08.090559       1 config-client.go:133] Send heartbeat: 2021-04-08 03:46:08.090539341 +0000 UTC m=+255.028128926
I0408 03:46:23.090362       1 config-client.go:133] Send heartbeat: 2021-04-08 03:46:23.090344426 +0000 UTC m=+270.027934021
I0408 03:46:38.090542       1 config-client.go:133] Send heartbeat: 2021-04-08 03:46:38.090524411 +0000 UTC m=+285.028113986
I0408 03:46:53.090349       1 config-client.go:133] Send heartbeat: 2021-04-08 03:46:53.090330706 +0000 UTC m=+300.027920271
I0408 03:47:08.090389       1 config-client.go:133] Send heartbeat: 2021-04-08 03:47:08.090372353 +0000 UTC m=+315.027961938
I0408 03:47:23.090383       1 config-client.go:133] Send heartbeat: 2021-04-08 03:47:23.090365688 +0000 UTC m=+330.027955283
I0408 03:47:38.090545       1 config-client.go:133] Send heartbeat: 2021-04-08 03:47:38.090526957 +0000 UTC m=+345.028116532
I0408 03:47:53.090541       1 config-client.go:133] Send heartbeat: 2021-04-08 03:47:53.090523383 +0000 UTC m=+360.028112978
I0408 03:48:08.090376       1 config-client.go:133] Send heartbeat: 2021-04-08 03:48:08.090352219 +0000 UTC m=+375.027941814
I0408 03:48:23.090566       1 config-client.go:133] Send heartbeat: 2021-04-08 03:48:23.090547824 +0000 UTC m=+390.028137389
I0408 03:48:38.090540       1 config-client.go:133] Send heartbeat: 2021-04-08 03:48:38.090520345 +0000 UTC m=+405.028109930
I0408 03:48:53.090386       1 config-client.go:133] Send heartbeat: 2021-04-08 03:48:53.090364341 +0000 UTC m=+420.027953936
I0408 03:49:08.090386       1 config-client.go:133] Send heartbeat: 2021-04-08 03:49:08.090367371 +0000 UTC m=+435.027956916
I0408 03:49:23.090357       1 config-client.go:133] Send heartbeat: 2021-04-08 03:49:23.090336605 +0000 UTC m=+450.027926100
I0408 03:49:38.090680       1 config-client.go:133] Send heartbeat: 2021-04-08 03:49:38.090661802 +0000 UTC m=+465.028251347
I0408 03:49:53.090381       1 config-client.go:133] Send heartbeat: 2021-04-08 03:49:53.090363113 +0000 UTC m=+480.027952688
I0408 03:50:08.090527       1 config-client.go:133] Send heartbeat: 2021-04-08 03:50:08.090507825 +0000 UTC m=+495.028097380
I0408 03:50:23.090518       1 config-client.go:133] Send heartbeat: 2021-04-08 03:50:23.090499894 +0000 UTC m=+510.028089489
I0408 03:50:38.090559       1 config-client.go:133] Send heartbeat: 2021-04-08 03:50:38.09054086 +0000 UTC m=+525.028130425
I0408 03:50:53.090338       1 config-client.go:133] Send heartbeat: 2021-04-08 03:50:53.090320467 +0000 UTC m=+540.027909962
I0408 03:51:08.090367       1 config-client.go:133] Send heartbeat: 2021-04-08 03:51:08.090349431 +0000 UTC m=+555.027938926
I0408 03:51:23.090369       1 config-client.go:133] Send heartbeat: 2021-04-08 03:51:23.09035098 +0000 UTC m=+570.027940585

kubeshare-node-daemon / gemini-scheduler

└> kubectl -n kube-system logs kubeshare-node-daemon-ln4wm -c gemini-scheduler

/usr/bin/nvidia-smi
[launcher] scheduler started on 0.0.0.0:49901
2021-04-08 03:41:53.571041 Gemini I/ There are 0 clients in the system...
2021-04-08 03:41:53.571168 Gemini I/ Monitor thread created.
2021-04-08 03:41:53.571180 Gemini I/ Waiting for incoming connection
2021-04-08 03:41:53.571196 Gemini I/ Watching '/kubeshare/scheduler/config'.
2021-04-08 03:42:34.165974 Gemini I/ File GPU-eec6cdf6-8303-ebfa-dc29-17b65addd74f modified with watch descriptor 1.
2021-04-08 03:42:34.165993 Gemini I/ Update containers' settings...
2021-04-08 03:42:34.166041 Gemini I/ There are 1 clients in the system...
2021-04-08 03:42:34.166080 Gemini I/ default/pod-test request: 0.10, limit: 0.50, memory limit: 1073741824 bytes
[launcher] pod manager id 'default/pod-test 50051' port '50051' start running
2021-04-08 03:42:34.169123 Gemini I/ Pod server port = 50051.
2021-04-08 03:42:34.169140 Gemini I/ scheduler 127.0.0.1:49901
2021-04-08 03:42:34.169230 Gemini I/ Received an incoming connection.
2021-04-08 03:42:34.169313 Gemini I/ GPU memory limit: 1073741824 bytes.

kubeshare-scheduler

└> kubectl -n kube-system logs kubeshare-scheduler                                            

W0408 03:41:50.694588       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0408 03:41:50.768486       1 controller.go:83] Creating event broadcaster
I0408 03:41:50.768651       1 controller.go:104] Setting up event handlers
I0408 03:41:50.768704       1 controller.go:126] Starting Foo controller
I0408 03:41:50.768711       1 controller.go:128] Waiting for informer caches to sync
I0408 03:41:50.768898       1 reflector.go:150] Starting reflector *v1.Node (30s) from pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105
I0408 03:41:50.768901       1 reflector.go:150] Starting reflector *v1.Pod (30s) from pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105
I0408 03:41:50.768918       1 reflector.go:185] Listing and watching *v1.Node from pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105
I0408 03:41:50.768922       1 reflector.go:185] Listing and watching *v1.Pod from pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105
I0408 03:41:50.768934       1 reflector.go:150] Starting reflector *v1.SharePod (30s) from pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105
I0408 03:41:50.768957       1 reflector.go:185] Listing and watching *v1.SharePod from pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105
I0408 03:41:50.868874       1 shared_informer.go:227] caches populated
I0408 03:41:50.868893       1 controller.go:133] Starting workers
I0408 03:41:50.868913       1 controller.go:142] Started workers
I0408 03:42:20.770545       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:42:20.772462       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:42:20.779372       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:42:31.975969       1 controller.go:268] SharePod 'default/pod-test' had been scheduled to node 'yb-server' GPUID 'cnjys'.
I0408 03:42:31.983826       1 controller.go:180] Successfully synced 'default/pod-test'
I0408 03:42:31.983937       1 event.go:281] Event(v1.ObjectReference{Kind:"SharePod", Namespace:"default", Name:"pod-test", UID:"d8e4e54a-34bd-4b23-8ef0-76c156e3b6ed", APIVersion:"kubeshare.nthu/v1", ResourceVersion:"548", FieldPath:""}): type: 'Normal' reason: 'Synced' SharePod scheduled successfully
I0408 03:42:50.770701       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:42:50.772581       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:42:50.779521       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:43:20.770828       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:43:20.772707       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:43:20.779664       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:43:50.770961       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:43:50.772828       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:43:50.779776       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:44:20.771112       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:44:20.772965       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:44:20.779935       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:44:50.771253       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:44:50.773071       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:44:50.780054       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:45:20.771372       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:45:20.773176       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:45:20.780174       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:45:50.771515       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:45:50.773295       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:45:50.780327       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:46:20.771654       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:46:20.773393       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:46:20.780520       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:46:50.771766       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:46:50.773503       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:46:50.780665       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:47:20.772088       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:47:20.773646       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:47:20.780792       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:47:50.772225       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:47:50.773768       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:47:50.781103       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:48:20.772397       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:48:20.773920       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:48:20.781274       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:48:50.772560       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:48:50.774067       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:48:50.781439       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:49:20.772814       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:49:20.774189       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:49:20.781560       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:49:50.772985       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:49:50.774312       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:49:50.781705       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:49:51.771081       1 reflector.go:418] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Watch close - *v1.SharePod total 13 items received
I0408 03:50:09.780213       1 reflector.go:418] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Watch close - *v1.Pod total 31 items received
I0408 03:50:20.773150       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:50:20.774433       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:50:20.781858       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:50:50.773287       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:50:50.774542       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:50:50.781971       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:51:20.773429       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:51:20.774667       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:51:20.782114       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:51:32.772933       1 reflector.go:418] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Watch close - *v1.Node total 13 items received
I0408 03:51:50.773582       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:51:50.774793       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:51:50.782268       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:52:20.773729       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:52:20.774890       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:52:20.782394       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:52:50.773851       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:52:50.775010       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:52:50.782578       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:53:20.773983       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:53:20.775108       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:53:20.782701       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:53:50.774113       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:53:50.775200       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:53:50.782819       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:54:20.774260       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:54:20.775329       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0408 03:54:20.783094       1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync

my pod_test

no logs

SirZen97 commented 3 years ago

I have fixed this problem.I changed the environment variable "POD_MANAGER_IP". The default value of "POD_MANAGER_IP" is wrong.I run command,"export POD_MANAGER_IP = $(head -n 1 /kubeshare/library/schedulerIP.txt)", in container to change the environment variable.

yanghaku commented 3 years ago

I have fixed this problem.I changed the environment variable "POD_MANAGER_IP". The default value of "POD_MANAGER_IP" is wrong.I run command,"export POD_MANAGER_IP = $(head -n 1 /kubeshare/library/schedulerIP.txt)", in container to change the environment variable.

Yes, it turned out to be this place. I also solved this problem, thank you!!!