search

hpe-storage / csi-driver

A Container Storage Interface (CSI) driver from HPE
https://scod.hpedev.io
Apache License 2.0
61 stars 56 forks source link

Error after upgrading from openshift 4.12 to 4.14 #418

Open saad0805 opened 1 month ago

saad0805 commented 1 month ago

Hello we are getting an error while starting pods with pvc after upgrading cluster from 4.12 to 4.14

Events: Type Reason Age From Message

Warning FailedMount 63m (x444 over 17h) kubelet MountVolume.SetUp failed for volume "pvc-a0d39123-fe95-408b-b83b-3cb153e30274" : rpc error: code = NotFound desc = rpc error: code = Unavailable desc = Failed to get storage provider from secrets, Post "http://primera3par-csp-svc:8080/containers/v1/tokens": dial tcp 172.30.56.180:8080: connect: no route to host Warning FailedMount 23m (x462 over 17h) kubelet MountVolume.SetUp failed for volume "pvc-6596fd6e-81f3-4610-8be9-b2faa91afce7" : rpc error: code = NotFound desc = rpc error: code = Unavailable desc = Failed to get storage provider from secrets, Post "http://primera3par-csp-svc:8080/containers/v1/tokens": dial tcp 172.30.56.180:8080: connect: no route to host Warning FailedMount 18m (x464 over 17h) kubelet MountVolume.SetUp failed for volume "pvc-92d693f6-d5b6-482e-b417-badec5d39cfe" : rpc error: code = NotFound desc = rpc error: code = Unavailable desc = Failed to get storage provider from secrets, Post "http://primera3par-csp-svc:8080/containers/v1/tokens": dial tcp 172.30.56.180:8080: connect: no route to host Warning FailedMount 12m (x78 over 17h) kubelet Unable to attach or mount volumes: unmounted volumes=[bpm-email-template-attachment bpm-filetransfer bpm-email-attachment], unattached volumes=[], failed to process volumes=[]: timed out waiting for the condition Warning FailedMount 3m52s (x335 over 17h) kubelet Unable to attach or mount volumes: unmounted volumes=[bpm-email-attachment bpm-email-template-attachment bpm-filetransfer], unattached volumes=[], failed to process volumes=[]: timed out waiting for the condition

but we dont have it on all worker nodes

we are using version 2.4.2 of openshift operator

datamattsson commented 1 month ago

You have a CNI/networking issue in the cluster. The CSI controller is unable to communicate with the CSP.

Restarting everything in hpe-storage sometimes help.

oc rollout restart -nhpe-storage ds,deploy --all