Closed Iamlovingit closed 4 years ago
Hi, could you provide the logs of both kubeshare-device-manager and crashed kubeshare-node-daemon?
kubectl -n kube-system logs kubeshare-device-manager
kubectl -n kube-system logs $(kubectl -n kube-system get pod -o name -l lsalab=kubeshare-node-daemon --field-selector 'status.phase=Failed') -c config-client
I had updated the README for more clarifying that docker and nvidia-docker2 version should < 19, because KubeShare restricts the GPU devices access in containers by adding a environment variable 'NVIDIA_VISIBLE_DEVICES' (https://github.com/NVIDIA/nvidia-container-runtime).
kubectl -n kube-system logs kubeshare-device-manager
Thanks for your reply. kubeshare-device-manager log
W0305 07:11:22.105403 1 client_config.go:543] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0305 07:11:22.128064 1 controller.go:89] Creating event broadcaster
I0305 07:11:22.128238 1 controller.go:106] Setting up event handlers
I0305 07:11:22.128279 1 controller.go:148] Starting SharePod controller
I0305 07:11:22.128289 1 controller.go:151] Waiting for informer caches to sync
I0305 07:11:22.128392 1 reflector.go:150] Starting reflector *v1.SharePod (30s) from pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105
I0305 07:11:22.128410 1 reflector.go:185] Listing and watching *v1.SharePod from pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105
I0305 07:11:22.128450 1 reflector.go:150] Starting reflector *v1.Pod (30s) from pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105
I0305 07:11:22.128475 1 reflector.go:185] Listing and watching *v1.Pod from pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105
I0305 07:11:22.157507 1 controller.go:417] Processing object: kube-state-metrics-5cb5c6986b-48crt
I0305 07:11:22.157540 1 controller.go:417] Processing object: kube-scheduler-k8s-master
I0305 07:11:22.157548 1 controller.go:417] Processing object: kubeshare-node-daemon-42n68
I0305 07:11:22.157558 1 controller.go:417] Processing object: hdfsmodel-flower-shawn-123-predictor-default-gpsgr-deploymklch4
I0305 07:11:22.157569 1 controller.go:417] Processing object: model-deploy-rnhkt-deployment-59c7594758-clslg
I0305 07:11:22.157576 1 controller.go:417] Processing object: speaker-99k5w
I0305 07:11:22.157582 1 controller.go:417] Processing object: speaker-d5ppj
I0305 07:11:22.157589 1 controller.go:417] Processing object: kubeshare-device-manager
I0305 07:11:22.157595 1 controller.go:417] Processing object: autoscaler-hpa-6746679d8d-vfn9h
I0305 07:11:22.157604 1 controller.go:417] Processing object: controller-6cd646bb5b-kpfsh
I0305 07:11:22.157676 1 controller.go:417] Processing object: eventing-controller-5d98dc9989-pk9zm
I0305 07:11:22.157709 1 controller.go:417] Processing object: elasticsearch-logging-0
I0305 07:11:22.157740 1 controller.go:417] Processing object: calico-node-vlns6
I0305 07:11:22.157762 1 controller.go:417] Processing object: kube-proxy-xdfsr
I0305 07:11:22.157789 1 controller.go:417] Processing object: config-map-hm57m-deployment-6b4f596bc4-kgbt5
I0305 07:11:22.157809 1 controller.go:417] Processing object: node-exporter-znmxp
I0305 07:11:22.157827 1 controller.go:417] Processing object: kube-controller-manager-k8s-master
I0305 07:11:22.157837 1 controller.go:417] Processing object: nfsmodel-flower-shawn-1-predictor-default-kqbr9-deployment4whpx
I0305 07:11:22.157879 1 controller.go:417] Processing object: kibana-logging-b5d75f556-vn2p2
I0305 07:11:22.157889 1 controller.go:417] Processing object: nfs-model-manager-sta5371918badac8f63d802cab9ff00049-deplopsjzq
I0305 07:11:22.157903 1 controller.go:417] Processing object: node-exporter-qgv6p
I0305 07:11:22.157911 1 controller.go:417] Processing object: zipkin-54b7b4cb87-glj7p
I0305 07:11:22.157918 1 controller.go:417] Processing object: kube-proxy-kcdns
I0305 07:11:22.157927 1 controller.go:417] Processing object: kube-proxy-z7jrj
I0305 07:11:22.157933 1 controller.go:417] Processing object: calico-node-hrq9k
I0305 07:11:22.157939 1 controller.go:417] Processing object: kubeshare-node-daemon-rqhkz
I0305 07:11:22.157946 1 controller.go:417] Processing object: istio-pilot-7c9d4ff745-jw4dg
I0305 07:11:22.157958 1 controller.go:417] Processing object: etcd-k8s-master
I0305 07:11:22.157965 1 controller.go:417] Processing object: networking-istio-b54f6b445-sj66x
I0305 07:11:22.157976 1 controller.go:417] Processing object: nvidia-device-plugin-daemonset-fph6c
I0305 07:11:22.157984 1 controller.go:417] Processing object: account-server-f5wnl-deployment-545c7489f6-ngdd9
I0305 07:11:22.157993 1 controller.go:417] Processing object: cluster-local-gateway-67c89b5dd9-2tkzw
I0305 07:11:22.158000 1 controller.go:417] Processing object: node-exporter-427zt
I0305 07:11:22.158013 1 controller.go:417] Processing object: kubeshare-scheduler
I0305 07:11:22.158025 1 controller.go:417] Processing object: nvidia-device-plugin-daemonset-mdbnw
I0305 07:11:22.158037 1 controller.go:417] Processing object: activator-5cb7c7f5b-jnhq2
I0305 07:11:22.158044 1 controller.go:417] Processing object: kubeshare-node-daemon-244bv
I0305 07:11:22.158054 1 controller.go:417] Processing object: imc-dispatcher-67dd977c6d-4vc7b
I0305 07:11:22.158062 1 controller.go:417] Processing object: model-manager-62pc9-deployment-6bfd5fd46b-4zvtx
I0305 07:11:22.158071 1 controller.go:417] Processing object: pytorch-cifar10-batching-predictor-default-dhhzk-deploymenkpflt
I0305 07:11:22.158081 1 controller.go:417] Processing object: istio-ingressgateway-75bd464c5d-42k88
I0305 07:11:22.158086 1 controller.go:417] Processing object: elasticsearch-logging-1
I0305 07:11:22.158094 1 controller.go:417] Processing object: controller-6d4c5459f6-5zvbq
I0305 07:11:22.158099 1 controller.go:417] Processing object: webhook-9474b654b-fscbq
I0305 07:11:22.158106 1 controller.go:417] Processing object: kube-apiserver-k8s-master
I0305 07:11:22.158111 1 controller.go:417] Processing object: imc-controller-676ccb778-8zrhh
I0305 07:11:22.158118 1 controller.go:417] Processing object: prometheus-system-1
I0305 07:11:22.158127 1 controller.go:417] Processing object: autoscaler-df6489797-9djjt
I0305 07:11:22.158134 1 controller.go:417] Processing object: calico-node-ms86k
I0305 07:11:22.158147 1 controller.go:417] Processing object: speaker-4ks66
I0305 07:11:22.158161 1 controller.go:417] Processing object: kfserving-controller-manager-0
I0305 07:11:22.158169 1 controller.go:417] Processing object: sources-controller-b8774f9cc-hjfjp
I0305 07:11:22.158175 1 controller.go:417] Processing object: coredns-5c98db65d4-k8c99
I0305 07:11:22.158181 1 controller.go:417] Processing object: coredns-5c98db65d4-mpxgk
I0305 07:11:22.158189 1 controller.go:417] Processing object: prometheus-system-0
I0305 07:11:22.158196 1 controller.go:417] Processing object: calico-kube-controllers-56cd854695-skt9j
I0305 07:11:22.158203 1 controller.go:417] Processing object: eventing-webhook-69d44558dd-f528k
I0305 07:11:22.158209 1 controller.go:417] Processing object: grafana-79bcd7c778-jkcsd
I0305 07:11:22.228425 1 shared_informer.go:227] caches populated
I0305 07:11:22.228521 1 controller.go:164] Starting workers
I0305 07:11:22.228534 1 controller.go:170] Started workers
I0305 07:11:22.228568 1 config.go:52] Start listening on 0.0.0.0:9797...
I0305 07:11:22.228873 1 config.go:64] Waiting for clients...
I0305 07:11:22.812127 1 controller.go:417] Processing object: kubeshare-device-manager
I0305 07:11:29.571366 1 controller.go:417] Processing object: kubeshare-node-daemon-rqhkz
I0305 07:11:52.130469 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:11:52.157599 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:11:59.749234 1 controller.go:417] Processing object: kubeshare-scheduler
I0305 07:12:22.130626 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:12:22.157877 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:12:52.130770 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:12:52.158146 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:13:22.130936 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:13:22.158408 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:13:28.347664 1 controller.go:417] Processing object: nfs-model-manager-sta5371918badac8f63d802cab9ff00049-deplopsjzq
I0305 07:13:29.409090 1 controller.go:417] Processing object: nfs-model-manager-sta5371918badac8f63d802cab9ff00049-deplopsjzq
I0305 07:13:41.545758 1 controller.go:417] Processing object: nfs-model-manager-sta5371918badac8f63d802cab9ff00049-deplopsjzq
I0305 07:13:52.131077 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:13:52.158696 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:14:21.494790 1 controller.go:417] Processing object: hdfsmodel-flower-shawn-123-predictor-default-gpsgr-deploymklch4
I0305 07:14:22.131234 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:14:22.159018 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:14:23.546408 1 controller.go:417] Processing object: hdfsmodel-flower-shawn-123-predictor-default-gpsgr-deploymklch4
I0305 07:14:35.697780 1 controller.go:417] Processing object: hdfsmodel-flower-shawn-123-predictor-default-gpsgr-deploymklch4
I0305 07:14:52.131382 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:14:52.159298 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:15:22.131532 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:15:22.159574 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:15:52.131696 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:15:52.159874 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:16:22.131864 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:16:22.160162 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:16:52.132006 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:16:52.160448 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:17:22.132165 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:17:22.160727 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:17:52.132326 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:17:52.161037 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:18:22.132499 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:18:22.161328 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:18:31.332341 1 controller.go:417] Processing object: nfs-model-manager-sta5371918badac8f63d802cab9ff00049-deplopsjzq
I0305 07:18:32.395458 1 controller.go:417] Processing object: nfs-model-manager-sta5371918badac8f63d802cab9ff00049-deplopsjzq
I0305 07:18:43.548932 1 controller.go:417] Processing object: nfs-model-manager-sta5371918badac8f63d802cab9ff00049-deplopsjzq
I0305 07:18:52.132638 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:18:52.161612 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:19:22.132812 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:19:22.161871 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:19:23.131723 1 reflector.go:418] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Watch close - *v1.SharePod total 0 items received
I0305 07:19:30.853420 1 controller.go:417] Processing object: hdfsmodel-flower-shawn-123-predictor-default-gpsgr-deploymklch4
I0305 07:19:31.870540 1 controller.go:417] Processing object: hdfsmodel-flower-shawn-123-predictor-default-gpsgr-deploymklch4
I0305 07:19:43.695402 1 controller.go:417] Processing object: hdfsmodel-flower-shawn-123-predictor-default-gpsgr-deploymklch4
I0305 07:19:52.132983 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:19:52.162163 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:20:22.133150 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:20:22.162464 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:20:52.133284 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:20:52.162714 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:21:04.158569 1 reflector.go:418] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Watch close - *v1.Pod total 15 items received
I0305 07:21:22.133417 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:21:22.162938 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:21:52.133590 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:21:52.163229 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:22:22.133762 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:22:22.163544 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:22:26.086090 1 controller.go:417] Processing object: kubeshare-node-daemon-42n68
I0305 07:22:52.133903 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:22:52.163858 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:23:22.134059 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:23:22.164112 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:23:42.736320 1 controller.go:417] Processing object: kubeshare-node-daemon-244bv
I0305 07:23:43.730258 1 controller.go:417] Processing object: nfs-model-manager-sta5371918badac8f63d802cab9ff00049-deplopsjzq
I0305 07:23:44.795910 1 controller.go:417] Processing object: nfs-model-manager-sta5371918badac8f63d802cab9ff00049-deplopsjzq
I0305 07:23:52.134216 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:23:52.164357 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:23:55.551845 1 controller.go:417] Processing object: nfs-model-manager-sta5371918badac8f63d802cab9ff00049-deplopsjzq
I0305 07:24:22.134348 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:24:22.164661 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:24:38.926482 1 controller.go:417] Processing object: hdfsmodel-flower-shawn-123-predictor-default-gpsgr-deploymklch4
I0305 07:24:39.961426 1 controller.go:417] Processing object: hdfsmodel-flower-shawn-123-predictor-default-gpsgr-deploymklch4
I0305 07:24:52.134486 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:24:52.164928 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:24:52.709432 1 controller.go:417] Processing object: hdfsmodel-flower-shawn-123-predictor-default-gpsgr-deploymklch4
I0305 07:25:19.133655 1 reflector.go:418] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Watch close - *v1.SharePod total 0 items received
I0305 07:25:22.134629 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:25:22.165200 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:25:51.203176 1 controller.go:417] Processing object: kubeshare-node-daemon-rqhkz
I0305 07:25:52.134775 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:25:52.165465 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:25:52.242963 1 controller.go:417] Processing object: kubeshare-node-daemon-rqhkz
I0305 07:26:12.636110 1 controller.go:417] Processing object: kubeshare-node-daemon-rqhkz
I0305 07:26:15.689546 1 controller.go:417] Processing object: kube-proxy-z7jrj
I0305 07:26:22.134934 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:26:22.165745 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:26:23.690228 1 controller.go:417] Processing object: kubeshare-node-daemon-rqhkz
I0305 07:26:25.511487 1 controller.go:417] Processing object: kubeshare-node-daemon-rqhkz
I0305 07:26:52.135091 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:26:52.166022 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:26:55.055653 1 controller.go:417] Processing object: kubeshare-node-daemon-rqhkz
I0305 07:26:56.689009 1 controller.go:417] Processing object: kube-proxy-z7jrj
I0305 07:27:06.690567 1 controller.go:417] Processing object: kubeshare-node-daemon-rqhkz
I0305 07:27:19.512421 1 controller.go:417] Processing object: kubeshare-node-daemon-rqhkz
I0305 07:27:22.135222 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:27:22.166272 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:27:22.689577 1 controller.go:417] Processing object: kube-proxy-z7jrj
I0305 07:27:35.687819 1 controller.go:417] Processing object: kube-proxy-z7jrj
I0305 07:27:50.062727 1 controller.go:417] Processing object: kubeshare-node-daemon-rqhkz
I0305 07:27:52.135381 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:27:52.166553 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:28:03.689683 1 controller.go:417] Processing object: kubeshare-node-daemon-rqhkz
I0305 07:28:22.135537 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:28:22.166838 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:28:25.160464 1 reflector.go:418] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Watch close - *v1.Pod total 22 items received
I0305 07:28:32.688537 1 controller.go:417] Processing object: kube-proxy-z7jrj
I0305 07:28:33.441485 1 controller.go:417] Processing object: kubeshare-node-daemon-rqhkz
I0305 07:28:45.688251 1 controller.go:417] Processing object: kube-proxy-z7jrj
I0305 07:28:45.983992 1 controller.go:417] Processing object: nfs-model-manager-sta5371918badac8f63d802cab9ff00049-deplopsjzq
I0305 07:28:47.042833 1 controller.go:417] Processing object: nfs-model-manager-sta5371918badac8f63d802cab9ff00049-deplopsjzq
I0305 07:28:52.135677 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:28:52.167136 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:28:57.883220 1 controller.go:417] Processing object: kubeshare-node-daemon-rqhkz
I0305 07:29:11.688815 1 controller.go:417] Processing object: kubeshare-node-daemon-rqhkz
I0305 07:29:22.135825 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:29:22.167416 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:29:45.751590 1 controller.go:417] Processing object: hdfsmodel-flower-shawn-123-predictor-default-gpsgr-deploymklch4
I0305 07:29:46.791622 1 controller.go:417] Processing object: hdfsmodel-flower-shawn-123-predictor-default-gpsgr-deploymklch4
I0305 07:29:52.135974 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:29:52.167713 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:30:00.690528 1 controller.go:417] Processing object: hdfsmodel-flower-shawn-123-predictor-default-gpsgr-deploymklch4
I0305 07:30:08.688195 1 controller.go:417] Processing object: kube-proxy-z7jrj
I0305 07:30:21.456837 1 controller.go:417] Processing object: kubeshare-node-daemon-rqhkz
I0305 07:30:22.136143 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:30:22.167994 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:30:22.687831 1 controller.go:417] Processing object: kube-proxy-z7jrj
I0305 07:30:52.031076 1 controller.go:417] Processing object: kubeshare-node-daemon-rqhkz
I0305 07:30:52.136298 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:30:52.168292 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:31:06.688057 1 controller.go:417] Processing object: kubeshare-node-daemon-rqhkz
I0305 07:31:22.136444 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:31:22.168580 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:31:52.136619 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:31:52.168876 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:32:22.136768 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:32:22.169160 1 reflector.go:268] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: forcing resync
I0305 07:32:50.135465 1 reflector.go:418] pkg/mod/k8s.io/client-go@v0.17.2/tools/cache/reflector.go:105: Watch close - *v1.SharePod total 0 items received
deamon log
2020/03/05 09:08:32 Loading NVML
Hi, could you provide the logs of both kubeshare-device-manager and crashed kubeshare-node-daemon? kubectl -n kube-system logs kubeshare-device-manager kubectl -n kube-system logs $(kubectl -n kube-system get pod -o name -l lsalab=kubeshare-node-daemon --field-selector 'status.phase=Failed') -c config-client
I had updated the README for more clarifying that docker and nvidia-docker2 version should < 19, because KubeShare restricts the GPU devices access in containers by adding a environment variable 'NVIDIA_VISIBLE_DEVICES' (https://github.com/NVIDIA/nvidia-container-runtime).
if the problem is dcoker version
, I'll try re-install my docker later.
Hi, but I think the problem should happened after creating a SharePod if the docker version is wrong. I should add a docker version check at initializing phase in the future. BTW, the node-daemon log you provided above seems like daemon was at initializing step. I think it will print some messages if NVML failed. Could you check if there is something more in daemon log? thanks!
Sorry for getting back to you late.
I have checked all three containers of kubeshare pod
. There is some message in gemini-scheduler
:
kubectl logs -n kube-system kubeshare-node-daemon-rqhkz -c gemini-scheduler
/usr/bin/nvidia-smi
[launcher] scheduler started on 0.0.0.0:49901
[launcher] scheduler started on 0.0.0.0:49902
2020-03-05 07:25:50.950682 Gemini I/ There are 0 clients in the system...
2020-03-05 07:25:50.950933 Gemini I/ Monitor thread created.
2020-03-05 07:25:50.950958 Gemini I/ Waiting for incoming connection
2020-03-05 07:25:50.950985 Gemini I/ Watching '/kubeshare/scheduler/config'.
[launcher] scheduler started on 0.0.0.0:49904
2020-03-05 07:25:50.952255 Gemini I/ There are 0 clients in the system...
2020-03-05 07:25:50.952538 Gemini I/ Monitor thread created.
2020-03-05 07:25:50.952560 Gemini I/ Waiting for incoming connection
2020-03-05 07:25:50.952584 Gemini I/ Watching '/kubeshare/scheduler/config'.
[launcher] scheduler started on 0.0.0.0:49903
2020-03-05 07:25:50.953586 Gemini I/ There are 0 clients in the system...
2020-03-05 07:25:50.953821 Gemini I/ Monitor thread created.
2020-03-05 07:25:50.953843 Gemini I/ Waiting for incoming connection
2020-03-05 07:25:50.953865 Gemini I/ Watching '/kubeshare/scheduler/config'.
2020-03-05 07:25:50.955070 Gemini I/ There are 0 clients in the system...
2020-03-05 07:25:50.955294 Gemini I/ Monitor thread created.
2020-03-05 07:25:50.955309 Gemini I/ Waiting for incoming connection
2020-03-05 07:25:50.955347 Gemini I/ Watching '/kubeshare/scheduler/config'.
At last, which docker
and nvidia-docker
version is your suggestion? Version 18
is ok?
Hi, the log from container gemini-scheduler
looks good.
The latest supported docker and nvidia-docker2 version for NVIDIA GPU device plugin is 18.09.7.
Hi, the log from container gemini-scheduler looks good. The latest supported docker and nvidia-docker2 version for NVIDIA GPU device plugin is 18.09.7.
Thanks, I install a lower docker, now it can work! Close.
hi, I followed the document to install the kubeshare, but the daemon pod in gpu node can not run. My docker version is
Is my docker version too high? which version can run?