1 slow ops, oldest one blocked for 1880444 sec, mon.e has slow ops

[vadlakiran]

Hi Team, we have installed rook-ceph in kubernetes cluster, and we are using hostbased deployment of rook-ceph, we have 3 workernodes attached the to the disk and recently we have issue on one of the workernode, we have restared the workernode.

after workernode has restared we can see the status below in rook-ceph, how can we resolve this issue

cluster: id: cc9da5b7-2e86-4abc-971a-701ce9b3d532 health: HEALTH_WARN 1 slow ops, oldest one blocked for 1880444 sec, mon.e has slow ops

services: mon: 3 daemons, quorum c,e,f (age 3w) mgr: a(active, since 3w) mds: 1/1 daemons up, 1 hot standby osd: 3 osds: 3 up (since 3w), 3 in (since 3w)

data: volumes: 1/1 healthy pools: 3 pools, 65 pgs objects: 3.05k objects, 516 MiB usage: 3.2 GiB used, 1.8 TiB / 1.8 TiB avail pgs: 65 active+clean

io: client: 1.2 KiB/s rd, 2 op/s rd, 0 op/s wr

Logs to submit:

• Operator's logs, if necessary

• Crashing pod(s) logs, if necessary

  To get logs, use kubectl -n <namespace> logs <pod name> When pasting logs, always surround them with backticks or use the insert code button from the Github UI. Read GitHub documentation if you need help.

Cluster Status to submit:

• Output of kubectl commands, if necessary

  To get the health of the cluster, use kubectl rook-ceph health To get the status of the cluster, use kubectl rook-ceph ceph status For more details, see the Rook kubectl Plugin

Environment:

• OS (e.g. from /etc/os-release): ubuntu
• Kernel (e.g. uname -a): Linux master1 4.15.0-206-generic #217-Ubuntu SMP Fri Feb 3 19:10:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
• Cloud provider or hardware configuration: its on premises
• Rook version (use rook version inside of a Rook Pod): v8.1.1
• Storage backend version (e.g. for ceph do ceph -v): v16.2.10
• Kubernetes version (use kubectl version):v1.21.5
• Kubernetes cluster type (e.g. Tectonic, GKE, OpenShift): on premises cluster

• Storage backend status (e.g. for Ceph use ceph health in the Rook Ceph toolbox): cluster: id: cc9da5b7-2e86-4abc-971a-701ce9b3d532 health: HEALTH_WARN 1 slow ops, oldest one blocked for 1880444 sec, mon.e has slow ops

  services: mon: 3 daemons, quorum c,e,f (age 3w) mgr: a(active, since 3w) mds: 1/1 daemons up, 1 hot standby osd: 3 osds: 3 up (since 3w), 3 in (since 3w)

  data: volumes: 1/1 healthy pools: 3 pools, 65 pgs objects: 3.05k objects, 516 MiB usage: 3.2 GiB used, 1.8 TiB / 1.8 TiB avail pgs: 65 active+clean

  io: client: 1.2 KiB/s rd, 2 op/s rd, 0 op/s wr

[vadlakiran]

please find the attached logs

[vadlakiran]

[Uploading mon-e.log…]()

[vadlakiran]

[Uploading operator1.log…]()

[vadlakiran]

attached the operator logs and mon-e logs please let me know how to resolve, its production issue, we have bloked, due to this we are not able to take backup of it, we have mounted pvc to this rook-ceph we are not able to go inside pvc from pod, getting permision denied due to this we are not able to take backup

please this is high priority issue

[travisn]

Are you saying this issue is preventing you from taking backups?

1 slow ops, oldest one blocked for 1880444 sec, mon.e has slow ops

It is just a health warning, so it is surprising if it is blocking the IO. If it is just mon.e with the problem, you could try scaling down that mon and fail it over to see if a new mon will help.

[vadlakiran]

@travisn yes due to this we are not able to take backup because we are using the persistant volumes to take backup, is there any way we can modify the permisions to access the PVC ?

[vadlakiran]

@travisn every 45 sec operator will monitor the mon's right it should be resolve by operator, why its not doing it, is there any thing i have to enable to self heal with operator ?

[vadlakiran]

@travisn could you please any advice on how we can enable to self heal with operator how we can access the mounted pvc backend using ceph storage we are not able to ls as well

[travisn]

You can try the direct mount tools

[travisn]

@travisn every 45 sec operator will monitor the mon's right it should be resolve by operator, why its not doing it, is there any thing i have to enable to self heal with operator ?

Are you referring to the mon failover? As long as mons are up, this is a different issue.

[vadlakiran]

@travisn yes, if mon fails automatically operator pod resolve the issue it should comeup right ?

regards mount issue: if i want to try with directmount tools , how i can confirm that specific workernode, we have issue with sepcific workernode, we are using sharedfilesys as type

[travisn]

@travisn yes, if mon fails automatically operator pod resolve the issue it should comeup right ?

Yes, a mon will failover by default after 10 min if it is out of quorum. The doc linked previously explains more.

regards mount issue: if i want to try with directmount tools , how i can confirm that specific workernode, we have issue with sepcific workernode, we are using sharedfilesys as type

If you want to mount it from a specific worker node, you could create the direct tools pod with a node selector.

[vadlakiran]

@travisn we can see below logs in the rbdplugin-provisioner, is this causing not able to access the mountpoint getting permision denied

`sudo kubectl logs csi-rbdplugin-provisioner-84c7fb8d76-q68lv -n rook-ceph csi-provisioner I0330 05:46:23.300526 1 csi-provisioner.go:138] Version: v3.0.0 I0330 05:46:23.300616 1 csi-provisioner.go:161] Building kube configs for running in cluster... I0330 05:46:26.237022 1 common.go:111] Probing CSI driver for readiness I0330 05:46:26.262363 1 csi-provisioner.go:277] CSI driver does not support PUBLISH_UNPUBLISH_VOLUME, not watching VolumeAttachments I0330 05:46:26.264594 1 leaderelection.go:248] attempting to acquire leader lease rook-ceph/rook-ceph-rbd-csi-ceph-com... E0402 11:59:38.713354 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": http2: client connection lost E0402 11:59:48.560986 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 11:59:57.772664 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:00:08.941466 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:00:21.230099 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:00:33.516702 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:00:46.924557 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:00:59.213800 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:01:19.596629 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:01:56.022719 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: i/o timeout E0402 12:02:03.724542 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:02:11.820950 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:02:22.163660 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:02:34.444823 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:02:45.613050 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:02:55.950080 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:03:05.165761 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:03:14.380690 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:03:23.596301 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:03:34.768516 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:03:48.172259 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:04:00.460616 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:04:09.677223 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:04:17.773506 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:04:26.988783 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:04:33.133365 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:04:40.400824 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:04:49.616259 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:04:58.829464 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:05:08.044470 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:05:20.332322 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:05:31.505712 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:05:40.721216 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:05:49.932707 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:05:59.148667 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:06:09.484996 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:06:17.580295 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:06:23.724642 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:06:30.988966 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:06:43.276684 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:06:51.375085 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:06:58.636941 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:07:09.804808 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:07:19.021395 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:07:29.360543 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:07:38.572435 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:07:47.788678 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:07:53.933206 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:08:03.149706 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:08:12.364779 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:08:23.532832 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:08:32.748570 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:08:52.396059 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:09:00.400787 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:09:07.756605 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:09:18.829735 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:09:24.972737 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:09:37.261601 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:09:47.692587 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:09:53.837106 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:10:03.052785 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:10:14.124800 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:10:26.412531 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:10:35.629243 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:10:46.060566 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:10:54.060405 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:11:00.208714 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:11:07.565208 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:11:18.637217 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:11:29.069262 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:11:35.212642 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:11:47.500755 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:11:55.500846 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:12:07.788130 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:12:17.004366 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:12:27.437062 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:12:36.652573 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:12:44.656821 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:12:52.013114 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:13:01.232566 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:13:07.372911 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:13:15.372713 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:13:22.733054 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:13:31.948449 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:13:39.953512 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:13:50.381272 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:13:58.381209 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:14:05.744735 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:14:18.028188 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:14:27.244434 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:14:36.461118 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:14:42.925195 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:14:50.892791 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:15:00.108824 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:15:13.644301 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:15:22.864233 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:15:33.904216 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:15:46.189274 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:15:56.652687 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:16:04.620458 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:16:21.228767 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:16:33.516896 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:16:41.488778 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:16:51.951549 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:17:01.169980 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:17:09.132360 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:17:21.420462 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:17:28.813211 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:17:39.853202 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:17:49.068529 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:18:01.356403 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:18:11.820531 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:18:21.036272 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:18:27.181333 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:18:35.149229 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:18:45.613478 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:18:53.580896 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:19:04.046448 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:19:13.264448 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:19:22.476297 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:19:31.692120 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:19:39.664726 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:19:45.804853 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:19:51.953883 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:19:58.092973 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:20:08.556717 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:20:14.701560 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:20:26.988600 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:20:38.028577 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:20:48.492900 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:20:59.536662 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:21:13.068234 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:21:21.040706 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:21:33.324472 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:21:40.716653 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:21:49.932762 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:21:57.900258 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:22:05.293232 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:22:14.508730 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:22:22.480354 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:22:29.868722 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:22:40.908943 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:22:50.124657 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:23:02.413056 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:23:09.805301 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:23:17.932702 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:23:28.236275 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:23:36.368000 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:23:46.668746 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:23:57.868865 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:24:07.085172 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:24:17.388420 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:24:26.608410 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:24:32.749756 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:24:40.876970 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:24:50.097049 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: no route to host E0402 12:25:00.481411 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:25:08.456317 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:25:18.669614 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:25:28.259832 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:25:34.996712 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:25:42.593671 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:25:49.077372 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:25:56.942969 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:26:07.221009 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:26:17.822336 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:26:22.960686 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:26:33.745738 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:26:42.524636 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:26:53.055473 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:26:59.788320 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:27:06.213512 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:27:16.988802 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:27:23.354661 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:27:32.476279 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:27:38.472413 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:27:45.142355 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:27:51.698397 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:28:01.039511 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:28:06.322808 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:28:16.884096 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:28:22.012813 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:28:27.383993 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:28:36.051936 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:28:42.110972 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:28:52.030314 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:29:01.565392 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:29:10.887716 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:29:18.493427 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:29:28.473617 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:29:37.330322 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:29:46.437647 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused E0402 12:29:56.948023 1 leaderelection.go:330] error retrieving resource lock rook-ceph/rook-ceph-rbd-csi-ceph-com: Get "https://10.233.0.1:443/apis/coordination.k8s.io/v1/namespaces/rook-ceph/leases/rook-ceph-rbd-csi-ceph-com": dial tcp 10.233.0.1:443: connect: connection refused

`

[vadlakiran]

@travisn rbdplugin-provisioner is mounting emptyDir, what it write in the emptyDir ?

[travisn]

Can you provide the following?

• What is the latest output of ceph status? to show if ceph is healthy
• Have you checked the CSI troubleshooting guide?
• The CSI provisioner is only used for new volumes. If you are having problems with existing volumes, it is not related to the provisioner.
• What exactly is the issue? IO from cephfs or rbd volumes is not working? Please summarize the issue, and perhaps @Rakshith-R can take a look if related to CSI.