search

stormshift / support

This repo should serve as a central source for reporting issues with stormshift
GNU General Public License v3.0
3 stars 0 forks source link

PVC stuck in pending - storageclass: ocs-external-storagecluster-cephfs #97

Closed rbo closed 2 years ago

rbo commented 2 years ago

Created test pvc

$ oc get pvc -n rbohne-storage-test test-pvc   -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    volume.beta.kubernetes.io/storage-provisioner: openshift-storage.cephfs.csi.ceph.com
    volume.kubernetes.io/storage-provisioner: openshift-storage.cephfs.csi.ceph.com
  creationTimestamp: "2022-07-10T10:03:27Z"
  finalizers:
  - kubernetes.io/pvc-protection
  name: test-pvc
  namespace: rbohne-storage-test
  resourceVersion: "156358543"
  uid: e67c1b12-3f28-4ffd-be53-c123c8d19b39
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: ocs-external-storagecluster-cephfs
  volumeMode: Filesystem
status:
  phase: Pending
$

Check logfiles:

$ oc logs csi-cephfsplugin-provisioner-c6c9bb974-8sgz8 --tail=100 -c csi-provisioner | grep test-pvc
I0710 10:08:05.540441       1 controller.go:1337] provision "rbohne-storage-test/test-pvc" class "ocs-external-storagecluster-cephfs": started
I0710 10:08:05.540623       1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"rbohne-storage-test", Name:"test-pvc", UID:"e67c1b12-3f28-4ffd-be53-c123c8d19b39", APIVersion:"v1", ResourceVersion:"156358543", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "rbohne-storage-test/test-pvc"
I0710 10:08:05.543622       1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"rbohne-storage-test", Name:"test-pvc", UID:"e67c1b12-3f28-4ffd-be53-c123c8d19b39", APIVersion:"v1", ResourceVersion:"156358543", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "ocs-external-storagecluster-cephfs": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 already exists
I0710 10:10:13.544703       1 controller.go:1337] provision "rbohne-storage-test/test-pvc" class "ocs-external-storagecluster-cephfs": started
I0710 10:10:13.544883       1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"rbohne-storage-test", Name:"test-pvc", UID:"e67c1b12-3f28-4ffd-be53-c123c8d19b39", APIVersion:"v1", ResourceVersion:"156358543", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "rbohne-storage-test/test-pvc"
I0710 10:10:13.548044       1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"rbohne-storage-test", Name:"test-pvc", UID:"e67c1b12-3f28-4ffd-be53-c123c8d19b39", APIVersion:"v1", ResourceVersion:"156358543", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "ocs-external-storagecluster-cephfs": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 already exists
$
rbo commented 2 years ago 
$ oc logs csi-cephfsplugin-provisioner-c6c9bb974-8sgz8 --tail=100 -c csi-cephfsplugin | grep pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39
I0710 10:05:58.496635       1 utils.go:191] ID: 4960 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 GRPC call: /csi.v1.Controller/CreateVolume
I0710 10:05:58.496755       1 utils.go:195] ID: 4960 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39","parameters":{"clusterID":"openshift-storage","fsName":"fs1","pool":"fs1data"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":5}}]}
E0710 10:05:58.496825       1 controllerserver.go:176] ID: 4960 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 an operation with the given Volume ID pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 already exists
E0710 10:05:58.496842       1 utils.go:200] ID: 4960 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 already exists
I0710 10:05:59.500484       1 utils.go:191] ID: 4961 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 GRPC call: /csi.v1.Controller/CreateVolume
I0710 10:05:59.500599       1 utils.go:195] ID: 4961 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39","parameters":{"clusterID":"openshift-storage","fsName":"fs1","pool":"fs1data"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":5}}]}
E0710 10:05:59.500676       1 controllerserver.go:176] ID: 4961 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 an operation with the given Volume ID pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 already exists
E0710 10:05:59.500696       1 utils.go:200] ID: 4961 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 already exists
I0710 10:06:01.523458       1 utils.go:191] ID: 4962 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 GRPC call: /csi.v1.Controller/CreateVolume
I0710 10:06:01.523598       1 utils.go:195] ID: 4962 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39","parameters":{"clusterID":"openshift-storage","fsName":"fs1","pool":"fs1data"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":5}}]}
E0710 10:06:01.523716       1 controllerserver.go:176] ID: 4962 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 an operation with the given Volume ID pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 already exists
E0710 10:06:01.523744       1 utils.go:200] ID: 4962 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 already exists
I0710 10:06:05.527654       1 utils.go:191] ID: 4963 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 GRPC call: /csi.v1.Controller/CreateVolume
I0710 10:06:05.527800       1 utils.go:195] ID: 4963 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39","parameters":{"clusterID":"openshift-storage","fsName":"fs1","pool":"fs1data"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":5}}]}
E0710 10:06:05.527879       1 controllerserver.go:176] ID: 4963 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 an operation with the given Volume ID pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 already exists
E0710 10:06:05.527901       1 utils.go:200] ID: 4963 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 already exists
I0710 10:06:13.531001       1 utils.go:191] ID: 4964 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 GRPC call: /csi.v1.Controller/CreateVolume
I0710 10:06:13.531108       1 utils.go:195] ID: 4964 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39","parameters":{"clusterID":"openshift-storage","fsName":"fs1","pool":"fs1data"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":5}}]}
E0710 10:06:13.531178       1 controllerserver.go:176] ID: 4964 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 an operation with the given Volume ID pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 already exists
E0710 10:06:13.531198       1 utils.go:200] ID: 4964 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 already exists
I0710 10:06:29.535276       1 utils.go:191] ID: 4965 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 GRPC call: /csi.v1.Controller/CreateVolume
I0710 10:06:29.535404       1 utils.go:195] ID: 4965 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39","parameters":{"clusterID":"openshift-storage","fsName":"fs1","pool":"fs1data"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":5}}]}
E0710 10:06:29.535486       1 controllerserver.go:176] ID: 4965 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 an operation with the given Volume ID pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 already exists
E0710 10:06:29.535509       1 utils.go:200] ID: 4965 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 already exists
I0710 10:07:01.539324       1 utils.go:191] ID: 4967 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 GRPC call: /csi.v1.Controller/CreateVolume
I0710 10:07:01.539458       1 utils.go:195] ID: 4967 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39","parameters":{"clusterID":"openshift-storage","fsName":"fs1","pool":"fs1data"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":5}}]}
E0710 10:07:01.539536       1 controllerserver.go:176] ID: 4967 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 an operation with the given Volume ID pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 already exists
E0710 10:07:01.539562       1 utils.go:200] ID: 4967 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 already exists
I0710 10:08:05.543150       1 utils.go:191] ID: 4974 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 GRPC call: /csi.v1.Controller/CreateVolume
I0710 10:08:05.543303       1 utils.go:195] ID: 4974 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39","parameters":{"clusterID":"openshift-storage","fsName":"fs1","pool":"fs1data"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":5}}]}
E0710 10:08:05.543415       1 controllerserver.go:176] ID: 4974 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 an operation with the given Volume ID pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 already exists
E0710 10:08:05.543435       1 utils.go:200] ID: 4974 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 already exists
I0710 10:10:13.547593       1 utils.go:191] ID: 4977 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 GRPC call: /csi.v1.Controller/CreateVolume
I0710 10:10:13.547731       1 utils.go:195] ID: 4977 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 GRPC request: {"capacity_range":{"required_bytes":1073741824},"name":"pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39","parameters":{"clusterID":"openshift-storage","fsName":"fs1","pool":"fs1data"},"secrets":"***stripped***","volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{"mode":5}}]}
E0710 10:10:13.547800       1 controllerserver.go:176] ID: 4977 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 an operation with the given Volume ID pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 already exists
E0710 10:10:13.547818       1 utils.go:200] ID: 4977 Req-ID: pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 GRPC error: rpc error: code = Aborted desc = an operation with the given Volume ID pvc-e67c1b12-3f28-4ffd-be53-c123c8d19b39 already exists
$
rbo commented 2 years ago

Let's restart some pods:

$ oc delete pods --wait=false -l app=csi-cephfsplugin-provisioner
pod "csi-cephfsplugin-provisioner-c6c9bb974-8sgz8" deleted
pod "csi-cephfsplugin-provisioner-c6c9bb974-gf5tm" deleted
$ oc delete pods --wait=false -l  app=csi-rbdplugin-provisioner
pod "csi-rbdplugin-provisioner-69f954d7cb-ftqcb" deleted
pod "csi-rbdplugin-provisioner-69f954d7cb-jzjf4" deleted
$ oc delete pods --wait=false -l  app=csi-rbdplugin
pod "csi-rbdplugin-2dkhp" deleted
pod "csi-rbdplugin-4pzmz" deleted
pod "csi-rbdplugin-4x2kd" deleted
$ oc delete pods --wait=false -l  app=csi-cephfslugin
No resources found
$ oc delete pods --wait=false -l  app=csi-cephfsplugin
pod "csi-cephfsplugin-gm5lv" deleted
pod "csi-cephfsplugin-jmzw2" deleted
pod "csi-cephfsplugin-m9pfc" deleted
$
rbo commented 2 years ago 
$ oc describe pvc/asdf
Name:          asdf
Namespace:     rbohne-storage-test
StorageClass:  ocs-external-storagecluster-cephfs
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner: openshift-storage.cephfs.csi.ceph.com
               volume.kubernetes.io/storage-provisioner: openshift-storage.cephfs.csi.ceph.com
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type     Reason                Age                   From                                                                                                                     Message
  ----     ------                ----                  ----                                                                                                                     -------
  Warning  ProvisioningFailed    22m                   openshift-storage.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-c6c9bb974-g52wp_ecba56b1-13ce-4c7e-a5ea-b09fdc8cdce8  failed to provision volume with StorageClass "ocs-external-storagecluster-cephfs": rpc error: code = DeadlineExceeded desc = context deadline exceeded
  Normal   ExternalProvisioning  4m36s (x84 over 24m)  persistentvolume-controller                                                                                              waiting for a volume to be created, either by external provisioner "openshift-storage.cephfs.csi.ceph.com" or manually created by system administrator
  Normal   Provisioning          3m40s (x14 over 24m)  openshift-storage.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-c6c9bb974-g52wp_ecba56b1-13ce-4c7e-a5ea-b09fdc8cdce8  External provisioner is provisioning volume for claim "rbohne-storage-test/asdf"
  Warning  ProvisioningFailed    3m40s (x13 over 22m)  openshift-storage.cephfs.csi.ceph.com_csi-cephfsplugin-provisioner-c6c9bb974-g52wp_ecba56b1-13ce-4c7e-a5ea-b09fdc8cdce8  failed to provision volume with StorageClass "ocs-external-storagecluster-cephfs": rpc error: code = Aborted desc = an operation with the given Volume ID pvc-d40d8164-cfd0-4019-9d2f-661decf29086 already exists
$
rbo commented 2 years ago

image https://access.redhat.com/articles/4731161 https://access.redhat.com/labs/odfsi/#T0RGIGFzIFNlbGYtTWFuYWdlZCBTZXJ2aWNlLDQuMTAuNCwwLDEsMCww

[root@ceph10 ~]# ceph -v
ceph version 16.2.0-152.el8cp (e456e8b705cb2f4a779689a0d80b122bcb0d67c9) pacific (stable)

We have to update the central ceph cluster.

Javatar81 commented 2 years ago

We have ODF 4.8 on OCP 4.9 cluster. My plan was to update ODF to 4.9 but I cannot because it is still in installing state (happened after the OCP 4.8 -> 4.9 update btw)

rbo commented 2 years ago

@Javatar81 ODF in internal mode is not effected. Non of the stormshift clusters are connected to the external coe ceph.

Ceph is updated to latest & greates version, pvc works again.