[cinder-csi-plugin] Unable to attach or mount volumes: timed out waiting for the condition

sylvainOL commented 3 years ago

/kind bug

What happened: Provisioner installed in our Kubernetes v1.20.7 cluster. The provisioner is able to create and attach PVCs to the required node but Pods using the PVCs enter a waiting state with the error Unable to attach or mount volumes: unmounted volumes=[data-volume], unattached volumes=[default-token-4w5lf data-volume]: timed out waiting for the condition

When the PVC is created a volume is created in OpenStack.
When the Pod is created the OpenStack volume is attached to the corresponding node.

How can I determine what Condition the Pod's waiting for, and what might be happening? I guess I've something wrong with my setup but can't figure out what.

What you expected to happen:

Provisioner creates volumes, attached them to the compute instance and Pod successfully mounts the volume.

How to reproduce it:

Using a simple PVC and Pod: -

---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: test
spec:
  accessModes:
  - ReadWriteOnce
  storageClassName: standard
  resources:
    requests:
      storage: '1Gi'

---
kind: Pod
apiVersion: v1
metadata:
  name: pysimple
spec:
  containers:
  - name: pysimple
    image: alanbchristie/pysimple:2019.8
    volumeMounts:
    - name: data-volume
      mountPath: /data
  volumes:
  - name: data-volume
    persistentVolumeClaim:
      claimName: test

Anything else we need to know?:

We're using kubespray release-2.16 to deploy this cluster. This is an automated deployment which was OK 3 weeks ago. I've reinstalled everything a second time and same issue appears

Environment:

cinder-csi-plugin version: 1.2.2@v1.20.0
OpenStack version: Wallaby
Others:
- kolla-ansible is used to deploy OpenStack. An upgrade has occured a month ago (before the previous reinstallation)
- istio is deployed on the setup but after cinder csi plugin

Logs

looking at #1206, I tried to retrieve / debug by myself but I may not gather the right logs :(

csi-cinder-controllerplugin csi-provisioner:

I0818 08:50:19.469540       1 feature_gate.go:243] feature gates: &{map[]}
I0818 08:50:19.469600       1 csi-provisioner.go:107] Version: v1.6.0-0-g321fa5c1c
I0818 08:50:19.469615       1 csi-provisioner.go:121] Building kube configs for running in cluster...
I0818 08:50:19.486309       1 connection.go:153] Connecting to unix:///var/lib/csi/sockets/pluginproxy/csi.sock
I0818 08:50:22.764956       1 common.go:111] Probing CSI driver for readiness
I0818 08:50:22.765002       1 connection.go:182] GRPC call: /csi.v1.Identity/Probe
I0818 08:50:22.765012       1 connection.go:183] GRPC request: {}
I0818 08:50:22.781511       1 connection.go:185] GRPC response: {}
I0818 08:50:22.781945       1 connection.go:186] GRPC error: <nil>
I0818 08:50:22.781957       1 connection.go:182] GRPC call: /csi.v1.Identity/GetPluginInfo
I0818 08:50:22.781961       1 connection.go:183] GRPC request: {}
I0818 08:50:22.783136       1 connection.go:185] GRPC response: {"name":"cinder.csi.openstack.org","vendor_version":"1.2.2@v1.20.0"}
I0818 08:50:22.783644       1 connection.go:186] GRPC error: <nil>
I0818 08:50:22.783669       1 csi-provisioner.go:163] Detected CSI driver cinder.csi.openstack.org
W0818 08:50:22.783681       1 metrics.go:142] metrics endpoint will not be started because `metrics-address` was not specified.
I0818 08:50:22.783721       1 connection.go:182] GRPC call: /csi.v1.Identity/GetPluginCapabilities
I0818 08:50:22.783737       1 connection.go:183] GRPC request: {}
I0818 08:50:22.785045       1 connection.go:185] GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"Service":{"type":2}}},{"Type":{"VolumeExpansion":{"type":1}}},{"Type":{"VolumeExpansion":{"type":2}}}]}
I0818 08:50:22.789692       1 connection.go:186] GRPC error: <nil>
I0818 08:50:22.789714       1 connection.go:182] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I0818 08:50:22.789887       1 connection.go:183] GRPC request: {}
I0818 08:50:22.791629       1 connection.go:185] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":6}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":10}}}]}
I0818 08:50:22.797457       1 connection.go:186] GRPC error: <nil>
I0818 08:50:22.797650       1 csi-provisioner.go:216] Supports migration from in-tree plugin: kubernetes.io/cinder
I0818 08:50:22.798069       1 controller.go:709] Using saving PVs to API server in background
I0818 08:50:22.798390       1 reflector.go:153] Starting reflector *v1.PersistentVolumeClaim (15m0s) from k8s.io/client-go/informers/factory.go:135
I0818 08:50:22.798511       1 reflector.go:188] Listing and watching *v1.PersistentVolumeClaim from k8s.io/client-go/informers/factory.go:135
I0818 08:50:22.798593       1 reflector.go:153] Starting reflector *v1.StorageClass (1h0m0s) from k8s.io/client-go/informers/factory.go:135
I0818 08:50:22.798617       1 reflector.go:188] Listing and watching *v1.StorageClass from k8s.io/client-go/informers/factory.go:135
I0818 08:50:22.898353       1 shared_informer.go:227] caches populated
I0818 08:50:22.898391       1 shared_informer.go:227] caches populated
I0818 08:50:22.898448       1 controller.go:799] Starting provisioner controller cinder.csi.openstack.org_csi-cinder-controllerplugin-58479c9dd8-wfzvn_ee188cb4-fe95-43af-8904-5e45eabd6a4a!
I0818 08:50:22.898488       1 clone_controller.go:58] Starting CloningProtection controller
I0818 08:50:22.898536       1 clone_controller.go:74] Started CloningProtection controller
I0818 08:50:22.898787       1 reflector.go:153] Starting reflector *v1.PersistentVolumeClaim (15m0s) from github.com/kubernetes-csi/external-provisioner/pkg/controller/clone_controller.go:72
I0818 08:50:22.898815       1 reflector.go:188] Listing and watching *v1.PersistentVolumeClaim from github.com/kubernetes-csi/external-provisioner/pkg/controller/clone_controller.go:72
I0818 08:50:22.898980       1 reflector.go:153] Starting reflector *v1.PersistentVolume (15m0s) from sigs.k8s.io/sig-storage-lib-external-provisioner/v5/controller/controller.go:833
I0818 08:50:22.898995       1 reflector.go:188] Listing and watching *v1.PersistentVolume from sigs.k8s.io/sig-storage-lib-external-provisioner/v5/controller/controller.go:833
I0818 08:50:22.898799       1 volume_store.go:97] Starting save volume queue
I0818 08:50:22.899358       1 reflector.go:153] Starting reflector *v1.StorageClass (15m0s) from sigs.k8s.io/sig-storage-lib-external-provisioner/v5/controller/controller.go:836
I0818 08:50:22.899724       1 reflector.go:188] Listing and watching *v1.StorageClass from sigs.k8s.io/sig-storage-lib-external-provisioner/v5/controller/controller.go:836
I0818 08:50:22.998802       1 shared_informer.go:227] caches populated
I0818 08:50:22.999389       1 controller.go:848] Started provisioner controller cinder.csi.openstack.org_csi-cinder-controllerplugin-58479c9dd8-wfzvn_ee188cb4-fe95-43af-8904-5e45eabd6a4a!
I0818 08:57:29.907350       1 reflector.go:432] github.com/kubernetes-csi/external-provisioner/pkg/controller/clone_controller.go:72: Watch close - *v1.PersistentVolumeClaim total 4 items received
I0818 08:57:33.906478       1 reflector.go:432] sigs.k8s.io/sig-storage-lib-external-provisioner/v5/controller/controller.go:836: Watch close - *v1.StorageClass total 0 items received
I0818 08:58:41.811870       1 reflector.go:432] k8s.io/client-go/informers/factory.go:135: Watch close - *v1.PersistentVolumeClaim total 10 items received
I0818 08:58:48.915698       1 reflector.go:432] sigs.k8s.io/sig-storage-lib-external-provisioner/v5/controller/controller.go:833: Watch close - *v1.PersistentVolume total 7 items received
I0818 09:00:04.810684       1 reflector.go:432] k8s.io/client-go/informers/factory.go:135: Watch close - *v1.StorageClass total 0 items received
I0818 09:02:48.911034       1 reflector.go:432] github.com/kubernetes-csi/external-provisioner/pkg/controller/clone_controller.go:72: Watch close - *v1.PersistentVolumeClaim total 10 items received
I0818 09:03:19.909183       1 reflector.go:432] sigs.k8s.io/sig-storage-lib-external-provisioner/v5/controller/controller.go:836: Watch close - *v1.StorageClass total 0 items received
I0818 09:04:10.815749       1 reflector.go:432] k8s.io/client-go/informers/factory.go:135: Watch close - *v1.PersistentVolumeClaim total 0 items received
I0818 09:05:18.918231       1 reflector.go:432] sigs.k8s.io/sig-storage-lib-external-provisioner/v5/controller/controller.go:833: Watch close - *v1.PersistentVolume total 6 items received
I0818 09:05:22.810262       1 reflector.go:278] k8s.io/client-go/informers/factory.go:135: forcing resync
I0818 09:05:22.904942       1 reflector.go:278] github.com/kubernetes-csi/external-provisioner/pkg/controller/clone_controller.go:72: forcing resync
I0818 09:05:22.913836       1 reflector.go:278] sigs.k8s.io/sig-storage-lib-external-provisioner/v5/controller/controller.go:833: forcing resync
I0818 09:07:38.813934       1 reflector.go:432] k8s.io/client-go/informers/factory.go:135: Watch close - *v1.StorageClass total 0 items received
I0818 09:09:23.913242       1 reflector.go:432] sigs.k8s.io/sig-storage-lib-external-provisioner/v5/controller/controller.go:836: Watch close - *v1.StorageClass total 0 items received
I0818 09:11:04.818935       1 reflector.go:432] k8s.io/client-go/informers/factory.go:135: Watch close - *v1.PersistentVolumeClaim total 0 items received
I0818 09:11:52.914087       1 reflector.go:432] github.com/kubernetes-csi/external-provisioner/pkg/controller/clone_controller.go:72: Watch close - *v1.PersistentVolumeClaim total 0 items received
I0818 09:11:53.921118       1 reflector.go:432] sigs.k8s.io/sig-storage-lib-external-provisioner/v5/controller/controller.go:833: Watch close - *v1.PersistentVolume total 0 items received
I0818 09:14:58.817419       1 reflector.go:432] k8s.io/client-go/informers/factory.go:135: Watch close - *v1.StorageClass total 0 items received
I0818 09:15:47.916971       1 reflector.go:432] sigs.k8s.io/sig-storage-lib-external-provisioner/v5/controller/controller.go:836: Watch close - *v1.StorageClass total 0 items received
I0818 09:17:31.822824       1 reflector.go:432] k8s.io/client-go/informers/factory.go:135: Watch close - *v1.PersistentVolumeClaim total 0 items received
I0818 09:17:58.923135       1 reflector.go:432] sigs.k8s.io/sig-storage-lib-external-provisioner/v5/controller/controller.go:833: Watch close - *v1.PersistentVolume total 0 items received
I0818 09:20:15.916920       1 reflector.go:432] github.com/kubernetes-csi/external-provisioner/pkg/controller/clone_controller.go:72: Watch close - *v1.PersistentVolumeClaim total 0 items received
I0818 09:20:22.810653       1 reflector.go:278] k8s.io/client-go/informers/factory.go:135: forcing resync
I0818 09:20:22.905278       1 reflector.go:278] github.com/kubernetes-csi/external-provisioner/pkg/controller/clone_controller.go:72: forcing resync
I0818 09:20:22.914184       1 reflector.go:278] sigs.k8s.io/sig-storage-lib-external-provisioner/v5/controller/controller.go:833: forcing resync
I0818 09:20:58.821762       1 reflector.go:432] k8s.io/client-go/informers/factory.go:135: Watch close - *v1.StorageClass total 0 items received

csi-cinder-controllerplugin csi-attacher:

I0818 08:50:18.397165       1 main.go:91] Version: v2.2.0-0-g97411fa7
I0818 08:50:18.398346       1 connection.go:153] Connecting to unix:///var/lib/csi/sockets/pluginproxy/csi.sock
I0818 08:50:22.538772       1 common.go:111] Probing CSI driver for readiness
I0818 08:50:22.538798       1 connection.go:182] GRPC call: /csi.v1.Identity/Probe
I0818 08:50:22.538803       1 connection.go:183] GRPC request: {}
I0818 08:50:22.547059       1 connection.go:185] GRPC response: {}
I0818 08:50:22.548361       1 connection.go:186] GRPC error: <nil>
I0818 08:50:22.548376       1 connection.go:182] GRPC call: /csi.v1.Identity/GetPluginInfo
I0818 08:50:22.548381       1 connection.go:183] GRPC request: {}
I0818 08:50:22.550971       1 connection.go:185] GRPC response: {"name":"cinder.csi.openstack.org","vendor_version":"1.2.2@v1.20.0"}
I0818 08:50:22.551859       1 connection.go:186] GRPC error: <nil>
I0818 08:50:22.551876       1 main.go:136] CSI driver name: "cinder.csi.openstack.org"
W0818 08:50:22.551891       1 metrics.go:142] metrics endpoint will not be started because `metrics-address` was not specified.
I0818 08:50:22.551906       1 connection.go:182] GRPC call: /csi.v1.Identity/GetPluginCapabilities
I0818 08:50:22.551918       1 connection.go:183] GRPC request: {}
I0818 08:50:22.553411       1 connection.go:185] GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"Service":{"type":2}}},{"Type":{"VolumeExpansion":{"type":1}}},{"Type":{"VolumeExpansion":{"type":2}}}]}
I0818 08:50:22.558042       1 connection.go:186] GRPC error: <nil>
I0818 08:50:22.558068       1 connection.go:182] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I0818 08:50:22.558084       1 connection.go:183] GRPC request: {}
I0818 08:50:22.559761       1 connection.go:185] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":6}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":10}}}]}
I0818 08:50:22.567832       1 connection.go:186] GRPC error: <nil>
I0818 08:50:22.567965       1 main.go:162] CSI driver supports ControllerPublishUnpublish, using real CSI handler
I0818 08:50:22.567987       1 connection.go:182] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I0818 08:50:22.567997       1 connection.go:183] GRPC request: {}
I0818 08:50:22.569444       1 connection.go:185] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":3}}},{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":2}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":6}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":7}}},{"Type":{"Rpc":{"type":10}}}]}
I0818 08:50:22.578677       1 connection.go:186] GRPC error: <nil>
I0818 08:50:22.578699       1 main.go:175] CSI driver supports list volumes published nodes. Using capability to reconcile volume attachment objects with actual backend state
I0818 08:50:22.578981       1 controller.go:121] Starting CSI attacher
I0818 08:50:22.579272       1 reflector.go:153] Starting reflector *v1.PersistentVolume (10m0s) from k8s.io/client-go/informers/factory.go:135
I0818 08:50:22.579300       1 reflector.go:188] Listing and watching *v1.PersistentVolume from k8s.io/client-go/informers/factory.go:135
I0818 08:50:22.579333       1 reflector.go:153] Starting reflector *v1beta1.VolumeAttachment (10m0s) from k8s.io/client-go/informers/factory.go:135
I0818 08:50:22.579353       1 reflector.go:188] Listing and watching *v1beta1.VolumeAttachment from k8s.io/client-go/informers/factory.go:135
I0818 08:50:22.579389       1 reflector.go:153] Starting reflector *v1beta1.CSINode (10m0s) from k8s.io/client-go/informers/factory.go:135
I0818 08:50:22.579404       1 reflector.go:188] Listing and watching *v1beta1.CSINode from k8s.io/client-go/informers/factory.go:135
I0818 08:50:22.679170       1 shared_informer.go:227] caches populated
I0818 08:50:22.679336       1 csi_handler.go:116] Reconciling VolumeAttachments with driver backend state
I0818 08:50:22.679381       1 connection.go:182] GRPC call: /csi.v1.Controller/ListVolumes
I0818 08:50:22.679402       1 connection.go:183] GRPC request: {}
I0818 08:50:23.102796       1 connection.go:185] GRPC response: {"entries":[{"status":{},"volume":{"capacity_bytes":1073741824,"volume_id":"8da5e62e-f083-49f8-abba-5b516ef533ef"}},{"status":{"published_node_ids":["73f4531b-1394-4481-8987-e317c5d86166"]},"volume":{"capacity_bytes":1073741824,"volume_id":"bd64dcc4-e4c8-43ae-a6e4-bf743bb0da7c"}},{"status":{"published_node_ids":["73f4531b-1394-4481-8987-e317c5d86166"]},"volume":{"capacity_bytes":1073741824,"volume_id":"5512538d-8144-4df4-9e42-86d948858dc8"}},{"status":{"published_node_ids":["73f4531b-1394-4481-8987-e317c5d86166"]},"volume":{"capacity_bytes":107374182400,"volume_id":"8ed4f75e-ec37-4c7a-8ef1-e7012f0b728e"}},{"status":{"published_node_ids":["73f4531b-1394-4481-8987-e317c5d86166"]},"volume":{"capacity_bytes":2147483648,"volume_id":"b8770c0d-f6e6-4eff-9f82-11ebac133597"}},{"status":{"published_node_ids":["2f579c81-5bd6-4eba-89de-3527d980963f"]},"volume":{"capacity_bytes":10737418240,"volume_id":"f1c7a8b9-dd2b-48bd-bf0d-584374976a9f"}},{"status":{"published_node_ids":["2f579c81-5bd6-4eba-89de-3527d980963f"]},"volume":{"capacity_bytes":32212254720,"volume_id":"e0f685ca-ca70-4f41-8f82-f26a852f47ba"}},{"status":{"published_node_ids":["2f579c81-5bd6-4eba-89de-3527d980963f"]},"volume":{"capacity_bytes":64424509440,"volume_id":"bbf570b4-03f5-4ad7-bf69-d8cb7e164612"}},{"status":{"published_node_ids":["73f4531b-1394-4481-8987-e317c5d86166"]},"volume":{"capacity_bytes":10737418240,"volume_id":"34e9ac1d-eeff-4a97-aa34-031611c4d7ab"}},{"status":{"published_node_ids":["73f4531b-1394-4481-8987-e317c5d86166"]},"volume":{"capacity_bytes":32212254720,"volume_id":"7f355dea-9e15-4a26-a28d-d665e923fc59"}},{"status":{"published_node_ids":["73f4531b-1394-4481-8987-e317c5d86166"]},"volume":{"capacity_bytes":64424509440,"volume_id":"5c18cbaf-c3d3-45ff-9fa2-9760870a47c1"}},{"status":{"published_node_ids":["df54ff75-264d-458f-a8bd-bff2f31e43df"]},"volume":{"capacity_bytes":10737418240,"volume_id":"b338d434-8841-4260-b420-63a20b9c6c7c"}},{"status":{"published_node_ids":["df54ff75-264d-458f-a8bd-bff2f31e43df"]},"volume":{"capacity_bytes":32212254720,"volume_id":"4ce782a4-d746-44af-81a8-d25a327c658e"}},{"status":{"published_node_ids":["df54ff75-264d-458f-a8bd-bff2f31e43df"]},"volume":{"capacity_bytes":64424509440,"volume_id":"41115f88-cc97-4cff-b985-7d1e78a531c7"}},{"status":{"published_node_ids":["9ff87456-3e69-4aa7-9755-2f1be362489f"]},"volume":{"capacity_bytes":10737418240,"volume_id":"56ed04b4-277b-464f-a656-c8eb8f77759e"}},{"status":{"published_node_ids":["9ff87456-3e69-4aa7-9755-2f1be362489f"]},"volume":{"capacity_bytes":32212254720,"volume_id":"2258e5c5-ec4b-4303-9a95-224cf590c7df"}},{"status":{"published_node_ids":["9ff87456-3e69-4aa7-9755-2f1be362489f"]},"volume":{"capacity_bytes":64424509440,"volume_id":"f983849c-3028-4cf6-b942-2a2aad168d4d"}},{"status":{"published_node_ids":["93799f27-b114-4f52-bf6a-d2cdd78873ec"]},"volume":{"capacity_bytes":10737418240,"volume_id":"a05b211b-bbf4-4116-8394-589a638d4ee4"}},{"status":{"published_node_ids":["93799f27-b114-4f52-bf6a-d2cdd78873ec"]},"volume":{"capacity_bytes":32212254720,"volume_id":"abc0cc35-a15f-44a8-b705-6c392d78f2f0"}},{"status":{"published_node_ids":["93799f27-b114-4f52-bf6a-d2cdd78873ec"]},"volume":{"capacity_bytes":64424509440,"volume_id":"2194f657-ce96-46ae-b150-e0266d0134d9"}},{"status":{"published_node_ids":["a45b3049-fd53-4c32-944e-a6ef20b9ca18"]},"volume":{"capacity_bytes":10737418240,"volume_id":"5dd5dc17-2625-4a3b-af36-4330ee419f3a"}},{"status":{"published_node_ids":["a45b3049-fd53-4c32-944e-a6ef20b9ca18"]},"volume":{"capacity_bytes":32212254720,"volume_id":"d6563d9e-75e7-42aa-b3ed-3574940dc469"}},{"status":{"published_node_ids":["a45b3049-fd53-4c32-944e-a6ef20b9ca18"]},"volume":{"capacity_bytes":64424509440,"volume_id":"8bd8b647-f185-4d6d-ae28-277581b66f57"}},{"status":{"published_node_ids":["4637f75c-c7d5-42e9-ba51-3f81f5e0a2db"]},"volume":{"capacity_bytes":1073741824,"volume_id":"0c59c9f0-4cde-4d89-bb99-afb32357b857"}},{"status":{"published_node_ids":["4637f75c-c7d5-42e9-ba51-3f81f5e0a2db"]},"volume":{"capacity_bytes":10737418240,"volume_id":"4334d94c-7fb8-40ea-a062-69380df03910"}},{"status":{"published_node_ids":["4637f75c-c7d5-42e9-ba51-3f81f5e0a2db"]},"volume":{"capacity_bytes":10737418240,"volume_id":"8b4d897a-1790-4808-94e8-c88c33476e5d"}},{"status":{"published_node_ids":["4637f75c-c7d5-42e9-ba51-3f81f5e0a2db"]},"volume":{"capacity_bytes":5368709120,"volume_id":"0a56df20-e7f8-497e-a0bf-402cc887b250"}},{"status":{"published_node_ids":["9586aff2-3182-453f-a29e-3eca9cd3cca9"]},"volume":{"capacity_bytes":1073741824,"volume_id":"78334dce-e15c-42e4-85a2-249001615ae8"}},{"status":{"published_node_ids":["9586aff2-3182-453f-a29e-3eca9cd3cca9"]},"volume":{"capacity_bytes":10737418240,"volume_id":"e5ba0852-cd74-4145-8598-cc22820f8bb4"}},{"status":{"published_node_ids":["9586aff2-3182-453f-a29e-3eca9cd3cca9"]},"volume":{"capacity_bytes":10737418240,"volume_id":"dc87d5dc-7e0c-4fe0-94e8-64e23fe7bc60"}},{"status":{"published_node_ids":["9586aff2-3182-453f-a29e-3eca9cd3cca9"]},"volume":{"capacity_bytes":5368709120,"volume_id":"d2044eda-1aeb-4b01-8c67-5b52dd755df4"}},{"status":{"published_node_ids":["f63040d4-427f-4203-b533-4b36cbdbc4f8"]},"volume":{"capacity_bytes":1073741824,"volume_id":"8c664e65-a6be-43f0-8369-de8ee08c3372"}},{"status":{"published_node_ids":["f63040d4-427f-4203-b533-4b36cbdbc4f8"]},"volume":{"capacity_bytes":10737418240,"volume_id":"afaa8327-85a2-415d-958e-52277081a8ff"}},{"status":{"published_node_ids":["f63040d4-427f-4203-b533-4b36cbdbc4f8"]},"volume":{"capacity_bytes":10737418240,"volume_id":"31830c99-4866-44ad-8875-19106c881092"}},{"status":{"published_node_ids":["f63040d4-427f-4203-b533-4b36cbdbc4f8"]},"volume":{"capacity_bytes":5368709120,"volume_id":"49b7c493-9961-42cc-9ff3-e85d50baf32c"}}]}
I0818 08:50:23.166384       1 connection.go:186] GRPC error: <nil>
I0818 08:51:23.166794       1 csi_handler.go:116] Reconciling VolumeAttachments with driver backend state
...
I0818 09:05:30.261071       1 csi_handler.go:116] Reconciling VolumeAttachments with driver backend state
I0818 09:05:30.261119       1 connection.go:182] GRPC call: /csi.v1.Controller/ListVolumes
I0818 09:05:30.261126       1 connection.go:183] GRPC request: {}
I0818 09:05:30.747998       1 connection.go:185] GRPC response: {"entries":[{"status":{"published_node_ids":["2f579c81-5bd6-4eba-89de-3527d980963f"]},"volume":{"capacity_bytes":1073741824,"volume_id":"485f7181-8fa5-458d-8f24-73bde6e698ff"}},{"status":{"published_node_ids":["73f4531b-1394-4481-8987-e317c5d86166"]},"volume":{"capacity_bytes":1073741824,"volume_id":"bd64dcc4-e4c8-43ae-a6e4-bf743bb0da7c"}},{"status":{"published_node_ids":["73f4531b-1394-4481-8987-e317c5d86166"]},"volume":{"capacity_bytes":1073741824,"volume_id":"5512538d-8144-4df4-9e42-86d948858dc8"}},{"status":{"published_node_ids":["73f4531b-1394-4481-8987-e317c5d86166"]},"volume":{"capacity_bytes":107374182400,"volume_id":"8ed4f75e-ec37-4c7a-8ef1-e7012f0b728e"}},{"status":{"published_node_ids":["73f4531b-1394-4481-8987-e317c5d86166"]},"volume":{"capacity_bytes":2147483648,"volume_id":"b8770c0d-f6e6-4eff-9f82-11ebac133597"}},{"status":{"published_node_ids":["2f579c81-5bd6-4eba-89de-3527d980963f"]},"volume":{"capacity_bytes":10737418240,"volume_id":"f1c7a8b9-dd2b-48bd-bf0d-584374976a9f"}},{"status":{"published_node_ids":["2f579c81-5bd6-4eba-89de-3527d980963f"]},"volume":{"capacity_bytes":32212254720,"volume_id":"e0f685ca-ca70-4f41-8f82-f26a852f47ba"}},{"status":{"published_node_ids":["2f579c81-5bd6-4eba-89de-3527d980963f"]},"volume":{"capacity_bytes":64424509440,"volume_id":"bbf570b4-03f5-4ad7-bf69-d8cb7e164612"}},{"status":{"published_node_ids":["73f4531b-1394-4481-8987-e317c5d86166"]},"volume":{"capacity_bytes":10737418240,"volume_id":"34e9ac1d-eeff-4a97-aa34-031611c4d7ab"}},{"status":{"published_node_ids":["73f4531b-1394-4481-8987-e317c5d86166"]},"volume":{"capacity_bytes":32212254720,"volume_id":"7f355dea-9e15-4a26-a28d-d665e923fc59"}},{"status":{"published_node_ids":["73f4531b-1394-4481-8987-e317c5d86166"]},"volume":{"capacity_bytes":64424509440,"volume_id":"5c18cbaf-c3d3-45ff-9fa2-9760870a47c1"}},{"status":{"published_node_ids":["df54ff75-264d-458f-a8bd-bff2f31e43df"]},"volume":{"capacity_bytes":10737418240,"volume_id":"b338d434-8841-4260-b420-63a20b9c6c7c"}},{"status":{"published_node_ids":["df54ff75-264d-458f-a8bd-bff2f31e43df"]},"volume":{"capacity_bytes":32212254720,"volume_id":"4ce782a4-d746-44af-81a8-d25a327c658e"}},{"status":{"published_node_ids":["df54ff75-264d-458f-a8bd-bff2f31e43df"]},"volume":{"capacity_bytes":64424509440,"volume_id":"41115f88-cc97-4cff-b985-7d1e78a531c7"}},{"status":{"published_node_ids":["9ff87456-3e69-4aa7-9755-2f1be362489f"]},"volume":{"capacity_bytes":10737418240,"volume_id":"56ed04b4-277b-464f-a656-c8eb8f77759e"}},{"status":{"published_node_ids":["9ff87456-3e69-4aa7-9755-2f1be362489f"]},"volume":{"capacity_bytes":32212254720,"volume_id":"2258e5c5-ec4b-4303-9a95-224cf590c7df"}},{"status":{"published_node_ids":["9ff87456-3e69-4aa7-9755-2f1be362489f"]},"volume":{"capacity_bytes":64424509440,"volume_id":"f983849c-3028-4cf6-b942-2a2aad168d4d"}},{"status":{"published_node_ids":["93799f27-b114-4f52-bf6a-d2cdd78873ec"]},"volume":{"capacity_bytes":10737418240,"volume_id":"a05b211b-bbf4-4116-8394-589a638d4ee4"}},{"status":{"published_node_ids":["93799f27-b114-4f52-bf6a-d2cdd78873ec"]},"volume":{"capacity_bytes":32212254720,"volume_id":"abc0cc35-a15f-44a8-b705-6c392d78f2f0"}},{"status":{"published_node_ids":["93799f27-b114-4f52-bf6a-d2cdd78873ec"]},"volume":{"capacity_bytes":64424509440,"volume_id":"2194f657-ce96-46ae-b150-e0266d0134d9"}},{"status":{"published_node_ids":["a45b3049-fd53-4c32-944e-a6ef20b9ca18"]},"volume":{"capacity_bytes":10737418240,"volume_id":"5dd5dc17-2625-4a3b-af36-4330ee419f3a"}},{"status":{"published_node_ids":["a45b3049-fd53-4c32-944e-a6ef20b9ca18"]},"volume":{"capacity_bytes":32212254720,"volume_id":"d6563d9e-75e7-42aa-b3ed-3574940dc469"}},{"status":{"published_node_ids":["a45b3049-fd53-4c32-944e-a6ef20b9ca18"]},"volume":{"capacity_bytes":64424509440,"volume_id":"8bd8b647-f185-4d6d-ae28-277581b66f57"}},{"status":{"published_node_ids":["4637f75c-c7d5-42e9-ba51-3f81f5e0a2db"]},"volume":{"capacity_bytes":1073741824,"volume_id":"0c59c9f0-4cde-4d89-bb99-afb32357b857"}},{"status":{"published_node_ids":["4637f75c-c7d5-42e9-ba51-3f81f5e0a2db"]},"volume":{"capacity_bytes":10737418240,"volume_id":"4334d94c-7fb8-40ea-a062-69380df03910"}},{"status":{"published_node_ids":["4637f75c-c7d5-42e9-ba51-3f81f5e0a2db"]},"volume":{"capacity_bytes":10737418240,"volume_id":"8b4d897a-1790-4808-94e8-c88c33476e5d"}},{"status":{"published_node_ids":["4637f75c-c7d5-42e9-ba51-3f81f5e0a2db"]},"volume":{"capacity_bytes":5368709120,"volume_id":"0a56df20-e7f8-497e-a0bf-402cc887b250"}},{"status":{"published_node_ids":["9586aff2-3182-453f-a29e-3eca9cd3cca9"]},"volume":{"capacity_bytes":1073741824,"volume_id":"78334dce-e15c-42e4-85a2-249001615ae8"}},{"status":{"published_node_ids":["9586aff2-3182-453f-a29e-3eca9cd3cca9"]},"volume":{"capacity_bytes":10737418240,"volume_id":"e5ba0852-cd74-4145-8598-cc22820f8bb4"}},{"status":{"published_node_ids":["9586aff2-3182-453f-a29e-3eca9cd3cca9"]},"volume":{"capacity_bytes":10737418240,"volume_id":"dc87d5dc-7e0c-4fe0-94e8-64e23fe7bc60"}},{"status":{"published_node_ids":["9586aff2-3182-453f-a29e-3eca9cd3cca9"]},"volume":{"capacity_bytes":5368709120,"volume_id":"d2044eda-1aeb-4b01-8c67-5b52dd755df4"}},{"status":{"published_node_ids":["f63040d4-427f-4203-b533-4b36cbdbc4f8"]},"volume":{"capacity_bytes":1073741824,"volume_id":"8c664e65-a6be-43f0-8369-de8ee08c3372"}},{"status":{"published_node_ids":["f63040d4-427f-4203-b533-4b36cbdbc4f8"]},"volume":{"capacity_bytes":10737418240,"volume_id":"afaa8327-85a2-415d-958e-52277081a8ff"}},{"status":{"published_node_ids":["f63040d4-427f-4203-b533-4b36cbdbc4f8"]},"volume":{"capacity_bytes":10737418240,"volume_id":"31830c99-4866-44ad-8875-19106c881092"}},{"status":{"published_node_ids":["f63040d4-427f-4203-b533-4b36cbdbc4f8"]},"volume":{"capacity_bytes":5368709120,"volume_id":"49b7c493-9961-42cc-9ff3-e85d50baf32c"}}]}
I0818 09:05:30.803159       1 connection.go:186] GRPC error: <nil>
I0818 09:05:34.597564       1 reflector.go:432] k8s.io/client-go/informers/factory.go:135: Watch close - *v1beta1.CSINode total 0 items received
I0818 09:05:48.599868       1 reflector.go:432] k8s.io/client-go/informers/factory.go:135: Watch close - *v1beta1.VolumeAttachment total 0 items received
I0818 09:06:30.803370       1 csi_handler.go:116] Reconciling VolumeAttachments with driver backend state
I0818 09:06:30.803435       1 connection.go:182] GRPC call: /csi.v1.Controller/ListVolumes
I0818 09:06:30.803448       1 connection.go:183] GRPC request: {}
I0818 09:06:31.218808       1 connection.go:185] GRPC response: {"entries":[{"status":{"published_node_ids":["2f579c81-5bd6-4eba-89de-3527d980963f"]},"volume":{"capacity_bytes":1073741824,"volume_id":"485f7181-8fa5-458d-8f24-73bde6e698ff"}},{"status":{"published_node_ids":["73f4531b-1394-4481-8987-e317c5d86166"]},"volume":{"capacity_bytes":1073741824,"volume_id":"bd64dcc4-e4c8-43ae-a6e4-bf743bb0da7c"}},{"status":{"published_node_ids":["73f4531b-1394-4481-8987-e317c5d86166"]},"volume":{"capacity_bytes":1073741824,"volume_id":"5512538d-8144-4df4-9e42-86d948858dc8"}},{"status":{"published_node_ids":["73f4531b-1394-4481-8987-e317c5d86166"]},"volume":{"capacity_bytes":107374182400,"volume_id":"8ed4f75e-ec37-4c7a-8ef1-e7012f0b728e"}},{"status":{"published_node_ids":["73f4531b-1394-4481-8987-e317c5d86166"]},"volume":{"capacity_bytes":2147483648,"volume_id":"b8770c0d-f6e6-4eff-9f82-11ebac133597"}},{"status":{"published_node_ids":["2f579c81-5bd6-4eba-89de-3527d980963f"]},"volume":{"capacity_bytes":10737418240,"volume_id":"f1c7a8b9-dd2b-48bd-bf0d-584374976a9f"}},{"status":{"published_node_ids":["2f579c81-5bd6-4eba-89de-3527d980963f"]},"volume":{"capacity_bytes":32212254720,"volume_id":"e0f685ca-ca70-4f41-8f82-f26a852f47ba"}},{"status":{"published_node_ids":["2f579c81-5bd6-4eba-89de-3527d980963f"]},"volume":{"capacity_bytes":64424509440,"volume_id":"bbf570b4-03f5-4ad7-bf69-d8cb7e164612"}},{"status":{"published_node_ids":["73f4531b-1394-4481-8987-e317c5d86166"]},"volume":{"capacity_bytes":10737418240,"volume_id":"34e9ac1d-eeff-4a97-aa34-031611c4d7ab"}},{"status":{"published_node_ids":["73f4531b-1394-4481-8987-e317c5d86166"]},"volume":{"capacity_bytes":32212254720,"volume_id":"7f355dea-9e15-4a26-a28d-d665e923fc59"}},{"status":{"published_node_ids":["73f4531b-1394-4481-8987-e317c5d86166"]},"volume":{"capacity_bytes":64424509440,"volume_id":"5c18cbaf-c3d3-45ff-9fa2-9760870a47c1"}},{"status":{"published_node_ids":["df54ff75-264d-458f-a8bd-bff2f31e43df"]},"volume":{"capacity_bytes":10737418240,"volume_id":"b338d434-8841-4260-b420-63a20b9c6c7c"}},{"status":{"published_node_ids":["df54ff75-264d-458f-a8bd-bff2f31e43df"]},"volume":{"capacity_bytes":32212254720,"volume_id":"4ce782a4-d746-44af-81a8-d25a327c658e"}},{"status":{"published_node_ids":["df54ff75-264d-458f-a8bd-bff2f31e43df"]},"volume":{"capacity_bytes":64424509440,"volume_id":"41115f88-cc97-4cff-b985-7d1e78a531c7"}},{"status":{"published_node_ids":["9ff87456-3e69-4aa7-9755-2f1be362489f"]},"volume":{"capacity_bytes":10737418240,"volume_id":"56ed04b4-277b-464f-a656-c8eb8f77759e"}},{"status":{"published_node_ids":["9ff87456-3e69-4aa7-9755-2f1be362489f"]},"volume":{"capacity_bytes":32212254720,"volume_id":"2258e5c5-ec4b-4303-9a95-224cf590c7df"}},{"status":{"published_node_ids":["9ff87456-3e69-4aa7-9755-2f1be362489f"]},"volume":{"capacity_bytes":64424509440,"volume_id":"f983849c-3028-4cf6-b942-2a2aad168d4d"}},{"status":{"published_node_ids":["93799f27-b114-4f52-bf6a-d2cdd78873ec"]},"volume":{"capacity_bytes":10737418240,"volume_id":"a05b211b-bbf4-4116-8394-589a638d4ee4"}},{"status":{"published_node_ids":["93799f27-b114-4f52-bf6a-d2cdd78873ec"]},"volume":{"capacity_bytes":32212254720,"volume_id":"abc0cc35-a15f-44a8-b705-6c392d78f2f0"}},{"status":{"published_node_ids":["93799f27-b114-4f52-bf6a-d2cdd78873ec"]},"volume":{"capacity_bytes":64424509440,"volume_id":"2194f657-ce96-46ae-b150-e0266d0134d9"}},{"status":{"published_node_ids":["a45b3049-fd53-4c32-944e-a6ef20b9ca18"]},"volume":{"capacity_bytes":10737418240,"volume_id":"5dd5dc17-2625-4a3b-af36-4330ee419f3a"}},{"status":{"published_node_ids":["a45b3049-fd53-4c32-944e-a6ef20b9ca18"]},"volume":{"capacity_bytes":32212254720,"volume_id":"d6563d9e-75e7-42aa-b3ed-3574940dc469"}},{"status":{"published_node_ids":["a45b3049-fd53-4c32-944e-a6ef20b9ca18"]},"volume":{"capacity_bytes":64424509440,"volume_id":"8bd8b647-f185-4d6d-ae28-277581b66f57"}},{"status":{"published_node_ids":["4637f75c-c7d5-42e9-ba51-3f81f5e0a2db"]},"volume":{"capacity_bytes":1073741824,"volume_id":"0c59c9f0-4cde-4d89-bb99-afb32357b857"}},{"status":{"published_node_ids":["4637f75c-c7d5-42e9-ba51-3f81f5e0a2db"]},"volume":{"capacity_bytes":10737418240,"volume_id":"4334d94c-7fb8-40ea-a062-69380df03910"}},{"status":{"published_node_ids":["4637f75c-c7d5-42e9-ba51-3f81f5e0a2db"]},"volume":{"capacity_bytes":10737418240,"volume_id":"8b4d897a-1790-4808-94e8-c88c33476e5d"}},{"status":{"published_node_ids":["4637f75c-c7d5-42e9-ba51-3f81f5e0a2db"]},"volume":{"capacity_bytes":5368709120,"volume_id":"0a56df20-e7f8-497e-a0bf-402cc887b250"}},{"status":{"published_node_ids":["9586aff2-3182-453f-a29e-3eca9cd3cca9"]},"volume":{"capacity_bytes":1073741824,"volume_id":"78334dce-e15c-42e4-85a2-249001615ae8"}},{"status":{"published_node_ids":["9586aff2-3182-453f-a29e-3eca9cd3cca9"]},"volume":{"capacity_bytes":10737418240,"volume_id":"e5ba0852-cd74-4145-8598-cc22820f8bb4"}},{"status":{"published_node_ids":["9586aff2-3182-453f-a29e-3eca9cd3cca9"]},"volume":{"capacity_bytes":10737418240,"volume_id":"dc87d5dc-7e0c-4fe0-94e8-64e23fe7bc60"}},{"status":{"published_node_ids":["9586aff2-3182-453f-a29e-3eca9cd3cca9"]},"volume":{"capacity_bytes":5368709120,"volume_id":"d2044eda-1aeb-4b01-8c67-5b52dd755df4"}},{"status":{"published_node_ids":["f63040d4-427f-4203-b533-4b36cbdbc4f8"]},"volume":{"capacity_bytes":1073741824,"volume_id":"8c664e65-a6be-43f0-8369-de8ee08c3372"}},{"status":{"published_node_ids":["f63040d4-427f-4203-b533-4b36cbdbc4f8"]},"volume":{"capacity_bytes":10737418240,"volume_id":"afaa8327-85a2-415d-958e-52277081a8ff"}},{"status":{"published_node_ids":["f63040d4-427f-4203-b533-4b36cbdbc4f8"]},"volume":{"capacity_bytes":10737418240,"volume_id":"31830c99-4866-44ad-8875-19106c881092"}},{"status":{"published_node_ids":["f63040d4-427f-4203-b533-4b36cbdbc4f8"]},"volume":{"capacity_bytes":5368709120,"volume_id":"49b7c493-9961-42cc-9ff3-e85d50baf32c"}}]}
I0818 09:06:31.279427       1 connection.go:186] GRPC error: <nil>
I0818 09:07:31.279729       1 csi_handler.go:116] Reconciling VolumeAttachments with driver backend state
I0818 09:07:31.279850       1 connection.go:182] GRPC call: /csi.v1.Controller/ListVolumes
I0818 09:07:31.279863       1 connection.go:183] GRPC request: {}
I0818 09:07:31.665018       1 connection.go:185] GRPC response: {"entries":[{"status":{"published_node_ids":["2f579c81-5bd6-4eba-89de-3527d980963f"]},"volume":{"capacity_bytes":1073741824,"volume_id":"485f7181-8fa5-458d-8f24-73bde6e698ff"}},{"status":{"published_node_ids":["73f4531b-1394-4481-8987-e317c5d86166"]},"volume":{"capacity_bytes":1073741824,"volume_id":"bd64dcc4-e4c8-43ae-a6e4-bf743bb0da7c"}},{"status":{"published_node_ids":["73f4531b-1394-4481-8987-e317c5d86166"]},"volume":{"capacity_bytes":1073741824,"volume_id":"5512538d-8144-4df4-9e42-86d948858dc8"}},{"status":{"published_node_ids":["73f4531b-1394-4481-8987-e317c5d86166"]},"volume":{"capacity_bytes":107374182400,"volume_id":"8ed4f75e-ec37-4c7a-8ef1-e7012f0b728e"}},{"status":{"published_node_ids":["73f4531b-1394-4481-8987-e317c5d86166"]},"volume":{"capacity_bytes":2147483648,"volume_id":"b8770c0d-f6e6-4eff-9f82-11ebac133597"}},{"status":{"published_node_ids":["2f579c81-5bd6-4eba-89de-3527d980963f"]},"volume":{"capacity_bytes":10737418240,"volume_id":"f1c7a8b9-dd2b-48bd-bf0d-584374976a9f"}},{"status":{"published_node_ids":["2f579c81-5bd6-4eba-89de-3527d980963f"]},"volume":{"capacity_bytes":32212254720,"volume_id":"e0f685ca-ca70-4f41-8f82-f26a852f47ba"}},{"status":{"published_node_ids":["2f579c81-5bd6-4eba-89de-3527d980963f"]},"volume":{"capacity_bytes":64424509440,"volume_id":"bbf570b4-03f5-4ad7-bf69-d8cb7e164612"}},{"status":{"published_node_ids":["73f4531b-1394-4481-8987-e317c5d86166"]},"volume":{"capacity_bytes":10737418240,"volume_id":"34e9ac1d-eeff-4a97-aa34-031611c4d7ab"}},{"status":{"published_node_ids":["73f4531b-1394-4481-8987-e317c5d86166"]},"volume":{"capacity_bytes":32212254720,"volume_id":"7f355dea-9e15-4a26-a28d-d665e923fc59"}},{"status":{"published_node_ids":["73f4531b-1394-4481-8987-e317c5d86166"]},"volume":{"capacity_bytes":64424509440,"volume_id":"5c18cbaf-c3d3-45ff-9fa2-9760870a47c1"}},{"status":{"published_node_ids":["df54ff75-264d-458f-a8bd-bff2f31e43df"]},"volume":{"capacity_bytes":10737418240,"volume_id":"b338d434-8841-4260-b420-63a20b9c6c7c"}},{"status":{"published_node_ids":["df54ff75-264d-458f-a8bd-bff2f31e43df"]},"volume":{"capacity_bytes":32212254720,"volume_id":"4ce782a4-d746-44af-81a8-d25a327c658e"}},{"status":{"published_node_ids":["df54ff75-264d-458f-a8bd-bff2f31e43df"]},"volume":{"capacity_bytes":64424509440,"volume_id":"41115f88-cc97-4cff-b985-7d1e78a531c7"}},{"status":{"published_node_ids":["9ff87456-3e69-4aa7-9755-2f1be362489f"]},"volume":{"capacity_bytes":10737418240,"volume_id":"56ed04b4-277b-464f-a656-c8eb8f77759e"}},{"status":{"published_node_ids":["9ff87456-3e69-4aa7-9755-2f1be362489f"]},"volume":{"capacity_bytes":32212254720,"volume_id":"2258e5c5-ec4b-4303-9a95-224cf590c7df"}},{"status":{"published_node_ids":["9ff87456-3e69-4aa7-9755-2f1be362489f"]},"volume":{"capacity_bytes":64424509440,"volume_id":"f983849c-3028-4cf6-b942-2a2aad168d4d"}},{"status":{"published_node_ids":["93799f27-b114-4f52-bf6a-d2cdd78873ec"]},"volume":{"capacity_bytes":10737418240,"volume_id":"a05b211b-bbf4-4116-8394-589a638d4ee4"}},{"status":{"published_node_ids":["93799f27-b114-4f52-bf6a-d2cdd78873ec"]},"volume":{"capacity_bytes":32212254720,"volume_id":"abc0cc35-a15f-44a8-b705-6c392d78f2f0"}},{"status":{"published_node_ids":["93799f27-b114-4f52-bf6a-d2cdd78873ec"]},"volume":{"capacity_bytes":64424509440,"volume_id":"2194f657-ce96-46ae-b150-e0266d0134d9"}},{"status":{"published_node_ids":["a45b3049-fd53-4c32-944e-a6ef20b9ca18"]},"volume":{"capacity_bytes":10737418240,"volume_id":"5dd5dc17-2625-4a3b-af36-4330ee419f3a"}},{"status":{"published_node_ids":["a45b3049-fd53-4c32-944e-a6ef20b9ca18"]},"volume":{"capacity_bytes":32212254720,"volume_id":"d6563d9e-75e7-42aa-b3ed-3574940dc469"}},{"status":{"published_node_ids":["a45b3049-fd53-4c32-944e-a6ef20b9ca18"]},"volume":{"capacity_bytes":64424509440,"volume_id":"8bd8b647-f185-4d6d-ae28-277581b66f57"}},{"status":{"published_node_ids":["4637f75c-c7d5-42e9-ba51-3f81f5e0a2db"]},"volume":{"capacity_bytes":1073741824,"volume_id":"0c59c9f0-4cde-4d89-bb99-afb32357b857"}},{"status":{"published_node_ids":["4637f75c-c7d5-42e9-ba51-3f81f5e0a2db"]},"volume":{"capacity_bytes":10737418240,"volume_id":"4334d94c-7fb8-40ea-a062-69380df03910"}},{"status":{"published_node_ids":["4637f75c-c7d5-42e9-ba51-3f81f5e0a2db"]},"volume":{"capacity_bytes":10737418240,"volume_id":"8b4d897a-1790-4808-94e8-c88c33476e5d"}},{"status":{"published_node_ids":["4637f75c-c7d5-42e9-ba51-3f81f5e0a2db"]},"volume":{"capacity_bytes":5368709120,"volume_id":"0a56df20-e7f8-497e-a0bf-402cc887b250"}},{"status":{"published_node_ids":["9586aff2-3182-453f-a29e-3eca9cd3cca9"]},"volume":{"capacity_bytes":1073741824,"volume_id":"78334dce-e15c-42e4-85a2-249001615ae8"}},{"status":{"published_node_ids":["9586aff2-3182-453f-a29e-3eca9cd3cca9"]},"volume":{"capacity_bytes":10737418240,"volume_id":"e5ba0852-cd74-4145-8598-cc22820f8bb4"}},{"status":{"published_node_ids":["9586aff2-3182-453f-a29e-3eca9cd3cca9"]},"volume":{"capacity_bytes":10737418240,"volume_id":"dc87d5dc-7e0c-4fe0-94e8-64e23fe7bc60"}},{"status":{"published_node_ids":["9586aff2-3182-453f-a29e-3eca9cd3cca9"]},"volume":{"capacity_bytes":5368709120,"volume_id":"d2044eda-1aeb-4b01-8c67-5b52dd755df4"}},{"status":{"published_node_ids":["f63040d4-427f-4203-b533-4b36cbdbc4f8"]},"volume":{"capacity_bytes":1073741824,"volume_id":"8c664e65-a6be-43f0-8369-de8ee08c3372"}},{"status":{"published_node_ids":["f63040d4-427f-4203-b533-4b36cbdbc4f8"]},"volume":{"capacity_bytes":10737418240,"volume_id":"afaa8327-85a2-415d-958e-52277081a8ff"}},{"status":{"published_node_ids":["f63040d4-427f-4203-b533-4b36cbdbc4f8"]},"volume":{"capacity_bytes":10737418240,"volume_id":"31830c99-4866-44ad-8875-19106c881092"}},{"status":{"published_node_ids":["f63040d4-427f-4203-b533-4b36cbdbc4f8"]},"volume":{"capacity_bytes":5368709120,"volume_id":"49b7c493-9961-42cc-9ff3-e85d50baf32c"}}]}
I0818 09:07:31.737239       1 connection.go:186] GRPC error: <nil>
I0818 09:08:30.597345       1 reflector.go:432] k8s.io/client-go/informers/factory.go:135: Watch close - *v1.PersistentVolume total 9 items received
I0818 09:08:31.738069       1 csi_handler.go:116] Reconciling VolumeAttachments with driver backend state
I0818 09:08:31.738219       1 connection.go:182] GRPC call: /csi.v1.Controller/ListVolumes

csi-cinder-controllerplugin cinder-csi-plugin:

I0818 08:50:21.489948       1 driver.go:69] Driver: cinder.csi.openstack.org
I0818 08:50:21.489998       1 driver.go:70] Driver version: 1.2.2@v1.20.0
I0818 08:50:21.490004       1 driver.go:71] CSI Spec version: 1.2.0
I0818 08:50:21.490009       1 driver.go:100] Enabling controller service capability: LIST_VOLUMES
I0818 08:50:21.490015       1 driver.go:100] Enabling controller service capability: CREATE_DELETE_VOLUME
I0818 08:50:21.490020       1 driver.go:100] Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME
I0818 08:50:21.490025       1 driver.go:100] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
I0818 08:50:21.490029       1 driver.go:100] Enabling controller service capability: LIST_SNAPSHOTS
I0818 08:50:21.490034       1 driver.go:100] Enabling controller service capability: EXPAND_VOLUME
I0818 08:50:21.490038       1 driver.go:100] Enabling controller service capability: CLONE_VOLUME
I0818 08:50:21.490043       1 driver.go:100] Enabling controller service capability: LIST_VOLUMES_PUBLISHED_NODES
I0818 08:50:21.490048       1 driver.go:112] Enabling volume access mode: SINGLE_NODE_WRITER
I0818 08:50:21.490054       1 driver.go:122] Enabling node service capability: STAGE_UNSTAGE_VOLUME
I0818 08:50:21.490059       1 driver.go:122] Enabling node service capability: EXPAND_VOLUME
I0818 08:50:21.490087       1 driver.go:122] Enabling node service capability: GET_VOLUME_STATS
I0818 08:50:21.490098       1 openstack.go:130] InitOpenStackProvider configFile: /etc/config/cloud.conf
I0818 08:50:21.490738       1 openstack.go:242] AuthURL: https://vim.pod4.opnfv.fr:5000
I0818 08:50:21.490757       1 openstack.go:243] UserID:
I0818 08:50:21.490764       1 openstack.go:244] Username: onap-master-service-mesh-ci
I0818 08:50:21.490770       1 openstack.go:245] TenantID: 4daf3b7e314d4be1b2d6bc24ee3b487d
I0818 08:50:21.490776       1 openstack.go:246] TenantName: onap-master-service-mesh
I0818 08:50:21.490783       1 openstack.go:247] TrustID:
I0818 08:50:21.490789       1 openstack.go:248] DomainID:
I0818 08:50:21.490795       1 openstack.go:249] DomainName: Default
I0818 08:50:21.490822       1 openstack.go:250] TenantDomainID:
I0818 08:50:21.490833       1 openstack.go:251] TenantDomainName:
I0818 08:50:21.490839       1 openstack.go:252] UserDomainID:
I0818 08:50:21.490845       1 openstack.go:253] UserDomainName:
I0818 08:50:21.490854       1 openstack.go:254] Region: RegionOne
I0818 08:50:21.490862       1 openstack.go:255] CAFile:
I0818 08:50:21.490870       1 openstack.go:256] CertFile:
I0818 08:50:21.490895       1 openstack.go:257] KeyFile:
I0818 08:50:21.490932       1 openstack.go:258] UseClouds: false
I0818 08:50:21.490946       1 openstack.go:259] CloudsFile:
I0818 08:50:21.490969       1 openstack.go:260] Cloud:
I0818 08:50:21.490978       1 openstack.go:261] ApplicationCredentialID:
I0818 08:50:21.491006       1 openstack.go:262] ApplicationCredentialName:
I0818 08:50:21.491013       1 openstack.go:88] Block storage opts: {0 false false}
I0818 08:50:21.491093       1 openstack.go:521] Using user-agent cinder-csi-plugin/v1.20.0 gophercloud/2.0.0
I0818 08:50:21.988224       1 mount_linux.go:188] Detected OS without systemd
I0818 08:50:21.989163       1 server.go:108] Listening for connections on address: &net.UnixAddr{Name:"/csi/csi.sock", Net:"unix"}
I0818 08:50:22.138555       1 utils.go:80] GRPC call: /csi.v1.Identity/Probe
I0818 08:50:22.138603       1 utils.go:81] GRPC request:
I0818 08:50:22.138638       1 identityserver.go:50] Probe() called with req
I0818 08:50:22.158993       1 utils.go:86] GRPC response:
I0818 08:50:22.160675       1 utils.go:80] GRPC call: /csi.v1.Identity/GetPluginInfo
I0818 08:50:22.160688       1 utils.go:81] GRPC request:
I0818 08:50:22.160700       1 identityserver.go:33] Using default GetPluginInfo
I0818 08:50:22.160705       1 utils.go:86] GRPC response: name:"cinder.csi.openstack.org" vendor_version:"1.2.2@v1.20.0"
I0818 08:50:22.161449       1 utils.go:80] GRPC call: /csi.v1.Identity/GetPluginCapabilities
I0818 08:50:22.161466       1 utils.go:81] GRPC request:
I0818 08:50:22.161481       1 identityserver.go:64] GetPluginCapabilities called with req
I0818 08:50:22.161489       1 utils.go:86] GRPC response: capabilities:<service:<type:CONTROLLER_SERVICE > > capabilities:<service:<type:VOLUME_ACCESSIBILITY_CONSTRAINTS > > capabilities:<volume_expansion:<type:ONLINE > > capabilities:<volume_expansion:<type:OFFLINE > >
I0818 08:50:22.162566       1 utils.go:80] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I0818 08:50:22.162602       1 utils.go:81] GRPC request:
I0818 08:50:22.162626       1 controllerserver.go:483] Using default ControllerGetCapabilities
I0818 08:50:22.162632       1 utils.go:86] GRPC response: capabilities:<rpc:<type:LIST_VOLUMES > > capabilities:<rpc:<type:CREATE_DELETE_VOLUME > > capabilities:<rpc:<type:PUBLISH_UNPUBLISH_VOLUME > > capabilities:<rpc:<type:CREATE_DELETE_SNAPSHOT > > capabilities:<rpc:<type:LIST_SNAPSHOTS > > capabilities:<rpc:<type:EXPAND_VOLUME > > capabilities:<rpc:<type:CLONE_VOLUME > > capabilities:<rpc:<type:LIST_VOLUMES_PUBLISHED_NODES > >
I0818 08:50:22.516118       1 utils.go:80] GRPC call: /csi.v1.Identity/GetPluginInfo
I0818 08:50:22.516142       1 utils.go:81] GRPC request:
I0818 08:50:22.516182       1 identityserver.go:33] Using default GetPluginInfo
I0818 08:50:22.516188       1 utils.go:86] GRPC response: name:"cinder.csi.openstack.org" vendor_version:"1.2.2@v1.20.0"
I0818 08:50:22.518643       1 utils.go:80] GRPC call: /csi.v1.Identity/Probe
I0818 08:50:22.518664       1 utils.go:81] GRPC request:
I0818 08:50:22.518691       1 identityserver.go:50] Probe() called with req
I0818 08:50:22.525167       1 utils.go:86] GRPC response:
I0818 08:50:22.525802       1 utils.go:80] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I0818 08:50:22.525823       1 utils.go:81] GRPC request:
I0818 08:50:22.525855       1 controllerserver.go:483] Using default ControllerGetCapabilities
I0818 08:50:22.525861       1 utils.go:86] GRPC response: capabilities:<rpc:<type:LIST_VOLUMES > > capabilities:<rpc:<type:CREATE_DELETE_VOLUME > > capabilities:<rpc:<type:PUBLISH_UNPUBLISH_VOLUME > > capabilities:<rpc:<type:CREATE_DELETE_SNAPSHOT > > capabilities:<rpc:<type:LIST_SNAPSHOTS > > capabilities:<rpc:<type:EXPAND_VOLUME > > capabilities:<rpc:<type:CLONE_VOLUME > > capabilities:<rpc:<type:LIST_VOLUMES_PUBLISHED_NODES > >
I0818 08:50:22.540690       1 utils.go:80] GRPC call: /csi.v1.Identity/Probe
I0818 08:50:22.540718       1 utils.go:81] GRPC request:
I0818 08:50:22.540755       1 identityserver.go:50] Probe() called with req
I0818 08:50:22.546511       1 utils.go:86] GRPC response:
I0818 08:50:22.550653       1 utils.go:80] GRPC call: /csi.v1.Identity/GetPluginInfo
I0818 08:50:22.550676       1 utils.go:81] GRPC request:
I0818 08:50:22.550717       1 identityserver.go:33] Using default GetPluginInfo
I0818 08:50:22.550726       1 utils.go:86] GRPC response: name:"cinder.csi.openstack.org" vendor_version:"1.2.2@v1.20.0"
I0818 08:50:22.552919       1 utils.go:80] GRPC call: /csi.v1.Identity/GetPluginCapabilities
I0818 08:50:22.552933       1 utils.go:81] GRPC request:
I0818 08:50:22.552961       1 identityserver.go:64] GetPluginCapabilities called with req
I0818 08:50:22.552972       1 utils.go:86] GRPC response: capabilities:<service:<type:CONTROLLER_SERVICE > > capabilities:<service:<type:VOLUME_ACCESSIBILITY_CONSTRAINTS > > capabilities:<volume_expansion:<type:ONLINE > > capabilities:<volume_expansion:<type:OFFLINE > >
I0818 08:50:22.559398       1 utils.go:80] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I0818 08:50:22.559423       1 utils.go:81] GRPC request:
I0818 08:50:22.559450       1 controllerserver.go:483] Using default ControllerGetCapabilities
I0818 08:50:22.559456       1 utils.go:86] GRPC response: capabilities:<rpc:<type:LIST_VOLUMES > > capabilities:<rpc:<type:CREATE_DELETE_VOLUME > > capabilities:<rpc:<type:PUBLISH_UNPUBLISH_VOLUME > > capabilities:<rpc:<type:CREATE_DELETE_SNAPSHOT > > capabilities:<rpc:<type:LIST_SNAPSHOTS > > capabilities:<rpc:<type:EXPAND_VOLUME > > capabilities:<rpc:<type:CLONE_VOLUME > > capabilities:<rpc:<type:LIST_VOLUMES_PUBLISHED_NODES > >
I0818 08:50:22.569075       1 utils.go:80] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I0818 08:50:22.569092       1 utils.go:81] GRPC request:
I0818 08:50:22.569121       1 controllerserver.go:483] Using default ControllerGetCapabilities
I0818 08:50:22.569127       1 utils.go:86] GRPC response: capabilities:<rpc:<type:LIST_VOLUMES > > capabilities:<rpc:<type:CREATE_DELETE_VOLUME > > capabilities:<rpc:<type:PUBLISH_UNPUBLISH_VOLUME > > capabilities:<rpc:<type:CREATE_DELETE_SNAPSHOT > > capabilities:<rpc:<type:LIST_SNAPSHOTS > > capabilities:<rpc:<type:EXPAND_VOLUME > > capabilities:<rpc:<type:CLONE_VOLUME > > capabilities:<rpc:<type:LIST_VOLUMES_PUBLISHED_NODES > >
I0818 08:50:22.680710       1 utils.go:80] GRPC call: /csi.v1.Controller/ListVolumes
I0818 08:50:22.680737       1 utils.go:81] GRPC request:
I0818 08:50:22.767819       1 utils.go:80] GRPC call: /csi.v1.Identity/Probe
I0818 08:50:22.767835       1 utils.go:81] GRPC request:
I0818 08:50:22.767871       1 identityserver.go:50] Probe() called with req
I0818 08:50:22.781095       1 utils.go:86] GRPC response:
I0818 08:50:22.782648       1 utils.go:80] GRPC call: /csi.v1.Identity/GetPluginInfo
I0818 08:50:22.782675       1 utils.go:81] GRPC request:
I0818 08:50:22.782858       1 identityserver.go:33] Using default GetPluginInfo
I0818 08:50:22.782871       1 utils.go:86] GRPC response: name:"cinder.csi.openstack.org" vendor_version:"1.2.2@v1.20.0"
I0818 08:50:22.784635       1 utils.go:80] GRPC call: /csi.v1.Identity/GetPluginCapabilities
I0818 08:50:22.784644       1 utils.go:81] GRPC request:
I0818 08:50:22.784664       1 identityserver.go:64] GetPluginCapabilities called with req
I0818 08:50:22.784671       1 utils.go:86] GRPC response: capabilities:<service:<type:CONTROLLER_SERVICE > > capabilities:<service:<type:VOLUME_ACCESSIBILITY_CONSTRAINTS > > capabilities:<volume_expansion:<type:ONLINE > > capabilities:<volume_expansion:<type:OFFLINE > >
I0818 08:50:22.791335       1 utils.go:80] GRPC call: /csi.v1.Controller/ControllerGetCapabilities
I0818 08:50:22.791354       1 utils.go:81] GRPC request:
I0818 08:50:22.791384       1 controllerserver.go:483] Using default ControllerGetCapabilities
I0818 08:50:22.791390       1 utils.go:86] GRPC response: capabilities:<rpc:<type:LIST_VOLUMES > > capabilities:<rpc:<type:CREATE_DELETE_VOLUME > > capabilities:<rpc:<type:PUBLISH_UNPUBLISH_VOLUME > > capabilities:<rpc:<type:CREATE_DELETE_SNAPSHOT > > capabilities:<rpc:<type:LIST_SNAPSHOTS > > capabilities:<rpc:<type:EXPAND_VOLUME > > capabilities:<rpc:<type:CLONE_VOLUME > > capabilities:<rpc:<type:LIST_VOLUMES_PUBLISHED_NODES > >
I0818 08:50:23.101653       1 utils.go:86] GRPC response: entries:<volume:<capacity_bytes:1073741824 volume_id:"8da5e62e-f083-49f8-abba-5b516ef533ef" > status:<> > entries:<volume:<capacity_bytes:1073741824 volume_id:"bd64dcc4-e4c8-43ae-a6e4-bf743bb0da7c" > status:<published_node_ids:"73f4531b-1394-4481-8987-e317c5d86166" > > entries:<volume:<capacity_bytes:1073741824 volume_id:"5512538d-8144-4df4-9e42-86d948858dc8" > status:<published_node_ids:"73f4531b-1394-4481-8987-e317c5d86166" > > entries:<volume:<capacity_bytes:107374182400 volume_id:"8ed4f75e-ec37-4c7a-8ef1-e7012f0b728e" > status:<published_node_ids:"73f4531b-1394-4481-8987-e317c5d86166" > > entries:<volume:<capacity_bytes:2147483648 volume_id:"b8770c0d-f6e6-4eff-9f82-11ebac133597" > status:<published_node_ids:"73f4531b-1394-4481-8987-e317c5d86166" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"f1c7a8b9-dd2b-48bd-bf0d-584374976a9f" > status:<published_node_ids:"2f579c81-5bd6-4eba-89de-3527d980963f" > > entries:<volume:<capacity_bytes:32212254720 volume_id:"e0f685ca-ca70-4f41-8f82-f26a852f47ba" > status:<published_node_ids:"2f579c81-5bd6-4eba-89de-3527d980963f" > > entries:<volume:<capacity_bytes:64424509440 volume_id:"bbf570b4-03f5-4ad7-bf69-d8cb7e164612" > status:<published_node_ids:"2f579c81-5bd6-4eba-89de-3527d980963f" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"34e9ac1d-eeff-4a97-aa34-031611c4d7ab" > status:<published_node_ids:"73f4531b-1394-4481-8987-e317c5d86166" > > entries:<volume:<capacity_bytes:32212254720 volume_id:"7f355dea-9e15-4a26-a28d-d665e923fc59" > status:<published_node_ids:"73f4531b-1394-4481-8987-e317c5d86166" > > entries:<volume:<capacity_bytes:64424509440 volume_id:"5c18cbaf-c3d3-45ff-9fa2-9760870a47c1" > status:<published_node_ids:"73f4531b-1394-4481-8987-e317c5d86166" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"b338d434-8841-4260-b420-63a20b9c6c7c" > status:<published_node_ids:"df54ff75-264d-458f-a8bd-bff2f31e43df" > > entries:<volume:<capacity_bytes:32212254720 volume_id:"4ce782a4-d746-44af-81a8-d25a327c658e" > status:<published_node_ids:"df54ff75-264d-458f-a8bd-bff2f31e43df" > > entries:<volume:<capacity_bytes:64424509440 volume_id:"41115f88-cc97-4cff-b985-7d1e78a531c7" > status:<published_node_ids:"df54ff75-264d-458f-a8bd-bff2f31e43df" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"56ed04b4-277b-464f-a656-c8eb8f77759e" > status:<published_node_ids:"9ff87456-3e69-4aa7-9755-2f1be362489f" > > entries:<volume:<capacity_bytes:32212254720 volume_id:"2258e5c5-ec4b-4303-9a95-224cf590c7df" > status:<published_node_ids:"9ff87456-3e69-4aa7-9755-2f1be362489f" > > entries:<volume:<capacity_bytes:64424509440 volume_id:"f983849c-3028-4cf6-b942-2a2aad168d4d" > status:<published_node_ids:"9ff87456-3e69-4aa7-9755-2f1be362489f" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"a05b211b-bbf4-4116-8394-589a638d4ee4" > status:<published_node_ids:"93799f27-b114-4f52-bf6a-d2cdd78873ec" > > entries:<volume:<capacity_bytes:32212254720 volume_id:"abc0cc35-a15f-44a8-b705-6c392d78f2f0" > status:<published_node_ids:"93799f27-b114-4f52-bf6a-d2cdd78873ec" > > entries:<volume:<capacity_bytes:64424509440 volume_id:"2194f657-ce96-46ae-b150-e0266d0134d9" > status:<published_node_ids:"93799f27-b114-4f52-bf6a-d2cdd78873ec" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"5dd5dc17-2625-4a3b-af36-4330ee419f3a" > status:<published_node_ids:"a45b3049-fd53-4c32-944e-a6ef20b9ca18" > > entries:<volume:<capacity_bytes:32212254720 volume_id:"d6563d9e-75e7-42aa-b3ed-3574940dc469" > status:<published_node_ids:"a45b3049-fd53-4c32-944e-a6ef20b9ca18" > > entries:<volume:<capacity_bytes:64424509440 volume_id:"8bd8b647-f185-4d6d-ae28-277581b66f57" > status:<published_node_ids:"a45b3049-fd53-4c32-944e-a6ef20b9ca18" > > entries:<volume:<capacity_bytes:1073741824 volume_id:"0c59c9f0-4cde-4d89-bb99-afb32357b857" > status:<published_node_ids:"4637f75c-c7d5-42e9-ba51-3f81f5e0a2db" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"4334d94c-7fb8-40ea-a062-69380df03910" > status:<published_node_ids:"4637f75c-c7d5-42e9-ba51-3f81f5e0a2db" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"8b4d897a-1790-4808-94e8-c88c33476e5d" > status:<published_node_ids:"4637f75c-c7d5-42e9-ba51-3f81f5e0a2db" > > entries:<volume:<capacity_bytes:5368709120 volume_id:"0a56df20-e7f8-497e-a0bf-402cc887b250" > status:<published_node_ids:"4637f75c-c7d5-42e9-ba51-3f81f5e0a2db" > > entries:<volume:<capacity_bytes:1073741824 volume_id:"78334dce-e15c-42e4-85a2-249001615ae8" > status:<published_node_ids:"9586aff2-3182-453f-a29e-3eca9cd3cca9" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"e5ba0852-cd74-4145-8598-cc22820f8bb4" > status:<published_node_ids:"9586aff2-3182-453f-a29e-3eca9cd3cca9" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"dc87d5dc-7e0c-4fe0-94e8-64e23fe7bc60" > status:<published_node_ids:"9586aff2-3182-453f-a29e-3eca9cd3cca9" > > entries:<volume:<capacity_bytes:5368709120 volume_id:"d2044eda-1aeb-4b01-8c67-5b52dd755df4" > status:<published_node_ids:"9586aff2-3182-453f-a29e-3eca9cd3cca9" > > entries:<volume:<capacity_bytes:1073741824 volume_id:"8c664e65-a6be-43f0-8369-de8ee08c3372" > status:<published_node_ids:"f63040d4-427f-4203-b533-4b36cbdbc4f8" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"afaa8327-85a2-415d-958e-52277081a8ff" > status:<published_node_ids:"f63040d4-427f-4203-b533-4b36cbdbc4f8" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"31830c99-4866-44ad-8875-19106c881092" > status:<published_node_ids:"f63040d4-427f-4203-b533-4b36cbdbc4f8" > > entries:<volume:<capacity_bytes:5368709120 volume_id:"49b7c493-9961-42cc-9ff3-e85d50baf32c" > status:<published_node_ids:"f63040d4-427f-4203-b533-4b36cbdbc4f8" > >
...
I0818 09:05:30.262056       1 utils.go:80] GRPC call: /csi.v1.Controller/ListVolumes
I0818 09:05:30.262682       1 utils.go:81] GRPC request:
I0818 09:05:30.744503       1 utils.go:86] GRPC response: entries:<volume:<capacity_bytes:1073741824 volume_id:"485f7181-8fa5-458d-8f24-73bde6e698ff" > status:<published_node_ids:"2f579c81-5bd6-4eba-89de-3527d980963f" > > entries:<volume:<capacity_bytes:1073741824 volume_id:"bd64dcc4-e4c8-43ae-a6e4-bf743bb0da7c" > status:<published_node_ids:"73f4531b-1394-4481-8987-e317c5d86166" > > entries:<volume:<capacity_bytes:1073741824 volume_id:"5512538d-8144-4df4-9e42-86d948858dc8" > status:<published_node_ids:"73f4531b-1394-4481-8987-e317c5d86166" > > entries:<volume:<capacity_bytes:107374182400 volume_id:"8ed4f75e-ec37-4c7a-8ef1-e7012f0b728e" > status:<published_node_ids:"73f4531b-1394-4481-8987-e317c5d86166" > > entries:<volume:<capacity_bytes:2147483648 volume_id:"b8770c0d-f6e6-4eff-9f82-11ebac133597" > status:<published_node_ids:"73f4531b-1394-4481-8987-e317c5d86166" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"f1c7a8b9-dd2b-48bd-bf0d-584374976a9f" > status:<published_node_ids:"2f579c81-5bd6-4eba-89de-3527d980963f" > > entries:<volume:<capacity_bytes:32212254720 volume_id:"e0f685ca-ca70-4f41-8f82-f26a852f47ba" > status:<published_node_ids:"2f579c81-5bd6-4eba-89de-3527d980963f" > > entries:<volume:<capacity_bytes:64424509440 volume_id:"bbf570b4-03f5-4ad7-bf69-d8cb7e164612" > status:<published_node_ids:"2f579c81-5bd6-4eba-89de-3527d980963f" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"34e9ac1d-eeff-4a97-aa34-031611c4d7ab" > status:<published_node_ids:"73f4531b-1394-4481-8987-e317c5d86166" > > entries:<volume:<capacity_bytes:32212254720 volume_id:"7f355dea-9e15-4a26-a28d-d665e923fc59" > status:<published_node_ids:"73f4531b-1394-4481-8987-e317c5d86166" > > entries:<volume:<capacity_bytes:64424509440 volume_id:"5c18cbaf-c3d3-45ff-9fa2-9760870a47c1" > status:<published_node_ids:"73f4531b-1394-4481-8987-e317c5d86166" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"b338d434-8841-4260-b420-63a20b9c6c7c" > status:<published_node_ids:"df54ff75-264d-458f-a8bd-bff2f31e43df" > > entries:<volume:<capacity_bytes:32212254720 volume_id:"4ce782a4-d746-44af-81a8-d25a327c658e" > status:<published_node_ids:"df54ff75-264d-458f-a8bd-bff2f31e43df" > > entries:<volume:<capacity_bytes:64424509440 volume_id:"41115f88-cc97-4cff-b985-7d1e78a531c7" > status:<published_node_ids:"df54ff75-264d-458f-a8bd-bff2f31e43df" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"56ed04b4-277b-464f-a656-c8eb8f77759e" > status:<published_node_ids:"9ff87456-3e69-4aa7-9755-2f1be362489f" > > entries:<volume:<capacity_bytes:32212254720 volume_id:"2258e5c5-ec4b-4303-9a95-224cf590c7df" > status:<published_node_ids:"9ff87456-3e69-4aa7-9755-2f1be362489f" > > entries:<volume:<capacity_bytes:64424509440 volume_id:"f983849c-3028-4cf6-b942-2a2aad168d4d" > status:<published_node_ids:"9ff87456-3e69-4aa7-9755-2f1be362489f" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"a05b211b-bbf4-4116-8394-589a638d4ee4" > status:<published_node_ids:"93799f27-b114-4f52-bf6a-d2cdd78873ec" > > entries:<volume:<capacity_bytes:32212254720 volume_id:"abc0cc35-a15f-44a8-b705-6c392d78f2f0" > status:<published_node_ids:"93799f27-b114-4f52-bf6a-d2cdd78873ec" > > entries:<volume:<capacity_bytes:64424509440 volume_id:"2194f657-ce96-46ae-b150-e0266d0134d9" > status:<published_node_ids:"93799f27-b114-4f52-bf6a-d2cdd78873ec" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"5dd5dc17-2625-4a3b-af36-4330ee419f3a" > status:<published_node_ids:"a45b3049-fd53-4c32-944e-a6ef20b9ca18" > > entries:<volume:<capacity_bytes:32212254720 volume_id:"d6563d9e-75e7-42aa-b3ed-3574940dc469" > status:<published_node_ids:"a45b3049-fd53-4c32-944e-a6ef20b9ca18" > > entries:<volume:<capacity_bytes:64424509440 volume_id:"8bd8b647-f185-4d6d-ae28-277581b66f57" > status:<published_node_ids:"a45b3049-fd53-4c32-944e-a6ef20b9ca18" > > entries:<volume:<capacity_bytes:1073741824 volume_id:"0c59c9f0-4cde-4d89-bb99-afb32357b857" > status:<published_node_ids:"4637f75c-c7d5-42e9-ba51-3f81f5e0a2db" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"4334d94c-7fb8-40ea-a062-69380df03910" > status:<published_node_ids:"4637f75c-c7d5-42e9-ba51-3f81f5e0a2db" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"8b4d897a-1790-4808-94e8-c88c33476e5d" > status:<published_node_ids:"4637f75c-c7d5-42e9-ba51-3f81f5e0a2db" > > entries:<volume:<capacity_bytes:5368709120 volume_id:"0a56df20-e7f8-497e-a0bf-402cc887b250" > status:<published_node_ids:"4637f75c-c7d5-42e9-ba51-3f81f5e0a2db" > > entries:<volume:<capacity_bytes:1073741824 volume_id:"78334dce-e15c-42e4-85a2-249001615ae8" > status:<published_node_ids:"9586aff2-3182-453f-a29e-3eca9cd3cca9" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"e5ba0852-cd74-4145-8598-cc22820f8bb4" > status:<published_node_ids:"9586aff2-3182-453f-a29e-3eca9cd3cca9" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"dc87d5dc-7e0c-4fe0-94e8-64e23fe7bc60" > status:<published_node_ids:"9586aff2-3182-453f-a29e-3eca9cd3cca9" > > entries:<volume:<capacity_bytes:5368709120 volume_id:"d2044eda-1aeb-4b01-8c67-5b52dd755df4" > status:<published_node_ids:"9586aff2-3182-453f-a29e-3eca9cd3cca9" > > entries:<volume:<capacity_bytes:1073741824 volume_id:"8c664e65-a6be-43f0-8369-de8ee08c3372" > status:<published_node_ids:"f63040d4-427f-4203-b533-4b36cbdbc4f8" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"afaa8327-85a2-415d-958e-52277081a8ff" > status:<published_node_ids:"f63040d4-427f-4203-b533-4b36cbdbc4f8" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"31830c99-4866-44ad-8875-19106c881092" > status:<published_node_ids:"f63040d4-427f-4203-b533-4b36cbdbc4f8" > > entries:<volume:<capacity_bytes:5368709120 volume_id:"49b7c493-9961-42cc-9ff3-e85d50baf32c" > status:<published_node_ids:"f63040d4-427f-4203-b533-4b36cbdbc4f8" > >
I0818 09:06:30.806764       1 utils.go:80] GRPC call: /csi.v1.Controller/ListVolumes
I0818 09:06:30.806814       1 utils.go:81] GRPC request:
I0818 09:06:31.215383       1 utils.go:86] GRPC response: entries:<volume:<capacity_bytes:1073741824 volume_id:"485f7181-8fa5-458d-8f24-73bde6e698ff" > status:<published_node_ids:"2f579c81-5bd6-4eba-89de-3527d980963f" > > entries:<volume:<capacity_bytes:1073741824 volume_id:"bd64dcc4-e4c8-43ae-a6e4-bf743bb0da7c" > status:<published_node_ids:"73f4531b-1394-4481-8987-e317c5d86166" > > entries:<volume:<capacity_bytes:1073741824 volume_id:"5512538d-8144-4df4-9e42-86d948858dc8" > status:<published_node_ids:"73f4531b-1394-4481-8987-e317c5d86166" > > entries:<volume:<capacity_bytes:107374182400 volume_id:"8ed4f75e-ec37-4c7a-8ef1-e7012f0b728e" > status:<published_node_ids:"73f4531b-1394-4481-8987-e317c5d86166" > > entries:<volume:<capacity_bytes:2147483648 volume_id:"b8770c0d-f6e6-4eff-9f82-11ebac133597" > status:<published_node_ids:"73f4531b-1394-4481-8987-e317c5d86166" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"f1c7a8b9-dd2b-48bd-bf0d-584374976a9f" > status:<published_node_ids:"2f579c81-5bd6-4eba-89de-3527d980963f" > > entries:<volume:<capacity_bytes:32212254720 volume_id:"e0f685ca-ca70-4f41-8f82-f26a852f47ba" > status:<published_node_ids:"2f579c81-5bd6-4eba-89de-3527d980963f" > > entries:<volume:<capacity_bytes:64424509440 volume_id:"bbf570b4-03f5-4ad7-bf69-d8cb7e164612" > status:<published_node_ids:"2f579c81-5bd6-4eba-89de-3527d980963f" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"34e9ac1d-eeff-4a97-aa34-031611c4d7ab" > status:<published_node_ids:"73f4531b-1394-4481-8987-e317c5d86166" > > entries:<volume:<capacity_bytes:32212254720 volume_id:"7f355dea-9e15-4a26-a28d-d665e923fc59" > status:<published_node_ids:"73f4531b-1394-4481-8987-e317c5d86166" > > entries:<volume:<capacity_bytes:64424509440 volume_id:"5c18cbaf-c3d3-45ff-9fa2-9760870a47c1" > status:<published_node_ids:"73f4531b-1394-4481-8987-e317c5d86166" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"b338d434-8841-4260-b420-63a20b9c6c7c" > status:<published_node_ids:"df54ff75-264d-458f-a8bd-bff2f31e43df" > > entries:<volume:<capacity_bytes:32212254720 volume_id:"4ce782a4-d746-44af-81a8-d25a327c658e" > status:<published_node_ids:"df54ff75-264d-458f-a8bd-bff2f31e43df" > > entries:<volume:<capacity_bytes:64424509440 volume_id:"41115f88-cc97-4cff-b985-7d1e78a531c7" > status:<published_node_ids:"df54ff75-264d-458f-a8bd-bff2f31e43df" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"56ed04b4-277b-464f-a656-c8eb8f77759e" > status:<published_node_ids:"9ff87456-3e69-4aa7-9755-2f1be362489f" > > entries:<volume:<capacity_bytes:32212254720 volume_id:"2258e5c5-ec4b-4303-9a95-224cf590c7df" > status:<published_node_ids:"9ff87456-3e69-4aa7-9755-2f1be362489f" > > entries:<volume:<capacity_bytes:64424509440 volume_id:"f983849c-3028-4cf6-b942-2a2aad168d4d" > status:<published_node_ids:"9ff87456-3e69-4aa7-9755-2f1be362489f" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"a05b211b-bbf4-4116-8394-589a638d4ee4" > status:<published_node_ids:"93799f27-b114-4f52-bf6a-d2cdd78873ec" > > entries:<volume:<capacity_bytes:32212254720 volume_id:"abc0cc35-a15f-44a8-b705-6c392d78f2f0" > status:<published_node_ids:"93799f27-b114-4f52-bf6a-d2cdd78873ec" > > entries:<volume:<capacity_bytes:64424509440 volume_id:"2194f657-ce96-46ae-b150-e0266d0134d9" > status:<published_node_ids:"93799f27-b114-4f52-bf6a-d2cdd78873ec" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"5dd5dc17-2625-4a3b-af36-4330ee419f3a" > status:<published_node_ids:"a45b3049-fd53-4c32-944e-a6ef20b9ca18" > > entries:<volume:<capacity_bytes:32212254720 volume_id:"d6563d9e-75e7-42aa-b3ed-3574940dc469" > status:<published_node_ids:"a45b3049-fd53-4c32-944e-a6ef20b9ca18" > > entries:<volume:<capacity_bytes:64424509440 volume_id:"8bd8b647-f185-4d6d-ae28-277581b66f57" > status:<published_node_ids:"a45b3049-fd53-4c32-944e-a6ef20b9ca18" > > entries:<volume:<capacity_bytes:1073741824 volume_id:"0c59c9f0-4cde-4d89-bb99-afb32357b857" > status:<published_node_ids:"4637f75c-c7d5-42e9-ba51-3f81f5e0a2db" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"4334d94c-7fb8-40ea-a062-69380df03910" > status:<published_node_ids:"4637f75c-c7d5-42e9-ba51-3f81f5e0a2db" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"8b4d897a-1790-4808-94e8-c88c33476e5d" > status:<published_node_ids:"4637f75c-c7d5-42e9-ba51-3f81f5e0a2db" > > entries:<volume:<capacity_bytes:5368709120 volume_id:"0a56df20-e7f8-497e-a0bf-402cc887b250" > status:<published_node_ids:"4637f75c-c7d5-42e9-ba51-3f81f5e0a2db" > > entries:<volume:<capacity_bytes:1073741824 volume_id:"78334dce-e15c-42e4-85a2-249001615ae8" > status:<published_node_ids:"9586aff2-3182-453f-a29e-3eca9cd3cca9" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"e5ba0852-cd74-4145-8598-cc22820f8bb4" > status:<published_node_ids:"9586aff2-3182-453f-a29e-3eca9cd3cca9" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"dc87d5dc-7e0c-4fe0-94e8-64e23fe7bc60" > status:<published_node_ids:"9586aff2-3182-453f-a29e-3eca9cd3cca9" > > entries:<volume:<capacity_bytes:5368709120 volume_id:"d2044eda-1aeb-4b01-8c67-5b52dd755df4" > status:<published_node_ids:"9586aff2-3182-453f-a29e-3eca9cd3cca9" > > entries:<volume:<capacity_bytes:1073741824 volume_id:"8c664e65-a6be-43f0-8369-de8ee08c3372" > status:<published_node_ids:"f63040d4-427f-4203-b533-4b36cbdbc4f8" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"afaa8327-85a2-415d-958e-52277081a8ff" > status:<published_node_ids:"f63040d4-427f-4203-b533-4b36cbdbc4f8" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"31830c99-4866-44ad-8875-19106c881092" > status:<published_node_ids:"f63040d4-427f-4203-b533-4b36cbdbc4f8" > > entries:<volume:<capacity_bytes:5368709120 volume_id:"49b7c493-9961-42cc-9ff3-e85d50baf32c" > status:<published_node_ids:"f63040d4-427f-4203-b533-4b36cbdbc4f8" > >
I0818 09:07:31.281619       1 utils.go:80] GRPC call: /csi.v1.Controller/ListVolumes
I0818 09:07:31.281680       1 utils.go:81] GRPC request:
I0818 09:07:31.664337       1 utils.go:86] GRPC response: entries:<volume:<capacity_bytes:1073741824 volume_id:"485f7181-8fa5-458d-8f24-73bde6e698ff" > status:<published_node_ids:"2f579c81-5bd6-4eba-89de-3527d980963f" > > entries:<volume:<capacity_bytes:1073741824 volume_id:"bd64dcc4-e4c8-43ae-a6e4-bf743bb0da7c" > status:<published_node_ids:"73f4531b-1394-4481-8987-e317c5d86166" > > entries:<volume:<capacity_bytes:1073741824 volume_id:"5512538d-8144-4df4-9e42-86d948858dc8" > status:<published_node_ids:"73f4531b-1394-4481-8987-e317c5d86166" > > entries:<volume:<capacity_bytes:107374182400 volume_id:"8ed4f75e-ec37-4c7a-8ef1-e7012f0b728e" > status:<published_node_ids:"73f4531b-1394-4481-8987-e317c5d86166" > > entries:<volume:<capacity_bytes:2147483648 volume_id:"b8770c0d-f6e6-4eff-9f82-11ebac133597" > status:<published_node_ids:"73f4531b-1394-4481-8987-e317c5d86166" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"f1c7a8b9-dd2b-48bd-bf0d-584374976a9f" > status:<published_node_ids:"2f579c81-5bd6-4eba-89de-3527d980963f" > > entries:<volume:<capacity_bytes:32212254720 volume_id:"e0f685ca-ca70-4f41-8f82-f26a852f47ba" > status:<published_node_ids:"2f579c81-5bd6-4eba-89de-3527d980963f" > > entries:<volume:<capacity_bytes:64424509440 volume_id:"bbf570b4-03f5-4ad7-bf69-d8cb7e164612" > status:<published_node_ids:"2f579c81-5bd6-4eba-89de-3527d980963f" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"34e9ac1d-eeff-4a97-aa34-031611c4d7ab" > status:<published_node_ids:"73f4531b-1394-4481-8987-e317c5d86166" > > entries:<volume:<capacity_bytes:32212254720 volume_id:"7f355dea-9e15-4a26-a28d-d665e923fc59" > status:<published_node_ids:"73f4531b-1394-4481-8987-e317c5d86166" > > entries:<volume:<capacity_bytes:64424509440 volume_id:"5c18cbaf-c3d3-45ff-9fa2-9760870a47c1" > status:<published_node_ids:"73f4531b-1394-4481-8987-e317c5d86166" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"b338d434-8841-4260-b420-63a20b9c6c7c" > status:<published_node_ids:"df54ff75-264d-458f-a8bd-bff2f31e43df" > > entries:<volume:<capacity_bytes:32212254720 volume_id:"4ce782a4-d746-44af-81a8-d25a327c658e" > status:<published_node_ids:"df54ff75-264d-458f-a8bd-bff2f31e43df" > > entries:<volume:<capacity_bytes:64424509440 volume_id:"41115f88-cc97-4cff-b985-7d1e78a531c7" > status:<published_node_ids:"df54ff75-264d-458f-a8bd-bff2f31e43df" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"56ed04b4-277b-464f-a656-c8eb8f77759e" > status:<published_node_ids:"9ff87456-3e69-4aa7-9755-2f1be362489f" > > entries:<volume:<capacity_bytes:32212254720 volume_id:"2258e5c5-ec4b-4303-9a95-224cf590c7df" > status:<published_node_ids:"9ff87456-3e69-4aa7-9755-2f1be362489f" > > entries:<volume:<capacity_bytes:64424509440 volume_id:"f983849c-3028-4cf6-b942-2a2aad168d4d" > status:<published_node_ids:"9ff87456-3e69-4aa7-9755-2f1be362489f" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"a05b211b-bbf4-4116-8394-589a638d4ee4" > status:<published_node_ids:"93799f27-b114-4f52-bf6a-d2cdd78873ec" > > entries:<volume:<capacity_bytes:32212254720 volume_id:"abc0cc35-a15f-44a8-b705-6c392d78f2f0" > status:<published_node_ids:"93799f27-b114-4f52-bf6a-d2cdd78873ec" > > entries:<volume:<capacity_bytes:64424509440 volume_id:"2194f657-ce96-46ae-b150-e0266d0134d9" > status:<published_node_ids:"93799f27-b114-4f52-bf6a-d2cdd78873ec" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"5dd5dc17-2625-4a3b-af36-4330ee419f3a" > status:<published_node_ids:"a45b3049-fd53-4c32-944e-a6ef20b9ca18" > > entries:<volume:<capacity_bytes:32212254720 volume_id:"d6563d9e-75e7-42aa-b3ed-3574940dc469" > status:<published_node_ids:"a45b3049-fd53-4c32-944e-a6ef20b9ca18" > > entries:<volume:<capacity_bytes:64424509440 volume_id:"8bd8b647-f185-4d6d-ae28-277581b66f57" > status:<published_node_ids:"a45b3049-fd53-4c32-944e-a6ef20b9ca18" > > entries:<volume:<capacity_bytes:1073741824 volume_id:"0c59c9f0-4cde-4d89-bb99-afb32357b857" > status:<published_node_ids:"4637f75c-c7d5-42e9-ba51-3f81f5e0a2db" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"4334d94c-7fb8-40ea-a062-69380df03910" > status:<published_node_ids:"4637f75c-c7d5-42e9-ba51-3f81f5e0a2db" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"8b4d897a-1790-4808-94e8-c88c33476e5d" > status:<published_node_ids:"4637f75c-c7d5-42e9-ba51-3f81f5e0a2db" > > entries:<volume:<capacity_bytes:5368709120 volume_id:"0a56df20-e7f8-497e-a0bf-402cc887b250" > status:<published_node_ids:"4637f75c-c7d5-42e9-ba51-3f81f5e0a2db" > > entries:<volume:<capacity_bytes:1073741824 volume_id:"78334dce-e15c-42e4-85a2-249001615ae8" > status:<published_node_ids:"9586aff2-3182-453f-a29e-3eca9cd3cca9" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"e5ba0852-cd74-4145-8598-cc22820f8bb4" > status:<published_node_ids:"9586aff2-3182-453f-a29e-3eca9cd3cca9" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"dc87d5dc-7e0c-4fe0-94e8-64e23fe7bc60" > status:<published_node_ids:"9586aff2-3182-453f-a29e-3eca9cd3cca9" > > entries:<volume:<capacity_bytes:5368709120 volume_id:"d2044eda-1aeb-4b01-8c67-5b52dd755df4" > status:<published_node_ids:"9586aff2-3182-453f-a29e-3eca9cd3cca9" > > entries:<volume:<capacity_bytes:1073741824 volume_id:"8c664e65-a6be-43f0-8369-de8ee08c3372" > status:<published_node_ids:"f63040d4-427f-4203-b533-4b36cbdbc4f8" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"afaa8327-85a2-415d-958e-52277081a8ff" > status:<published_node_ids:"f63040d4-427f-4203-b533-4b36cbdbc4f8" > > entries:<volume:<capacity_bytes:10737418240 volume_id:"31830c99-4866-44ad-8875-19106c881092" > status:<published_node_ids:"f63040d4-427f-4203-b533-4b36cbdbc4f8" > > entries:<volume:<capacity_bytes:5368709120 volume_id:"49b7c493-9961-42cc-9ff3-e85d50baf32c" > status:<published_node_ids:"f63040d4-427f-4203-b533-4b36cbdbc4f8" > >
I0818 09:08:31.740862       1 utils.go:80] GRPC call: /csi.v1.Controller/ListVolumes
I0818 09:08:31.740955       1 utils.go:81] GRPC request:

csi-cinder-nodeplugin cinder-csi-plugin of used node:

I0818 08:25:22.170423       1 driver.go:69] Driver: cinder.csi.openstack.org
I0818 08:25:22.170486       1 driver.go:70] Driver version: 1.2.2@v1.20.0
I0818 08:25:22.170492       1 driver.go:71] CSI Spec version: 1.2.0
I0818 08:25:22.170500       1 driver.go:100] Enabling controller service capability: LIST_VOLUMES
I0818 08:25:22.170506       1 driver.go:100] Enabling controller service capability: CREATE_DELETE_VOLUME
I0818 08:25:22.170509       1 driver.go:100] Enabling controller service capability: PUBLISH_UNPUBLISH_VOLUME
I0818 08:25:22.170513       1 driver.go:100] Enabling controller service capability: CREATE_DELETE_SNAPSHOT
I0818 08:25:22.170517       1 driver.go:100] Enabling controller service capability: LIST_SNAPSHOTS
I0818 08:25:22.170520       1 driver.go:100] Enabling controller service capability: EXPAND_VOLUME
I0818 08:25:22.170524       1 driver.go:100] Enabling controller service capability: CLONE_VOLUME
I0818 08:25:22.170527       1 driver.go:100] Enabling controller service capability: LIST_VOLUMES_PUBLISHED_NODES
I0818 08:25:22.170531       1 driver.go:112] Enabling volume access mode: SINGLE_NODE_WRITER
I0818 08:25:22.170535       1 driver.go:122] Enabling node service capability: STAGE_UNSTAGE_VOLUME
I0818 08:25:22.170539       1 driver.go:122] Enabling node service capability: EXPAND_VOLUME
I0818 08:25:22.170543       1 driver.go:122] Enabling node service capability: GET_VOLUME_STATS
I0818 08:25:22.170548       1 openstack.go:130] InitOpenStackProvider configFile: /etc/config/cloud.conf
I0818 08:25:22.170966       1 openstack.go:242] AuthURL: https://vim.pod4.opnfv.fr:5000
I0818 08:25:22.170983       1 openstack.go:243] UserID:
I0818 08:25:22.170989       1 openstack.go:244] Username: onap-master-service-mesh-ci
I0818 08:25:22.170993       1 openstack.go:245] TenantID: 4daf3b7e314d4be1b2d6bc24ee3b487d
I0818 08:25:22.170998       1 openstack.go:246] TenantName: onap-master-service-mesh
I0818 08:25:22.171003       1 openstack.go:247] TrustID:
I0818 08:25:22.171008       1 openstack.go:248] DomainID:
I0818 08:25:22.171017       1 openstack.go:249] DomainName: Default
I0818 08:25:22.171024       1 openstack.go:250] TenantDomainID:
I0818 08:25:22.171029       1 openstack.go:251] TenantDomainName:
I0818 08:25:22.171034       1 openstack.go:252] UserDomainID:
I0818 08:25:22.171039       1 openstack.go:253] UserDomainName:
I0818 08:25:22.171045       1 openstack.go:254] Region: RegionOne
I0818 08:25:22.171050       1 openstack.go:255] CAFile:
I0818 08:25:22.171056       1 openstack.go:256] CertFile:
I0818 08:25:22.171061       1 openstack.go:257] KeyFile:
I0818 08:25:22.171067       1 openstack.go:258] UseClouds: false
I0818 08:25:22.171073       1 openstack.go:259] CloudsFile:
I0818 08:25:22.171079       1 openstack.go:260] Cloud:
I0818 08:25:22.171086       1 openstack.go:261] ApplicationCredentialID:
I0818 08:25:22.171092       1 openstack.go:262] ApplicationCredentialName:
I0818 08:25:22.171096       1 openstack.go:88] Block storage opts: {0 false false}
I0818 08:25:22.171141       1 openstack.go:521] Using user-agent cinder-csi-plugin/v1.20.0 gophercloud/2.0.0
I0818 08:25:22.641185       1 mount_linux.go:188] Detected OS without systemd
I0818 08:25:22.643012       1 server.go:108] Listening for connections on address: &net.UnixAddr{Name:"/csi/csi.sock", Net:"unix"}
I0818 08:25:23.041966       1 utils.go:80] GRPC call: /csi.v1.Identity/GetPluginInfo
I0818 08:25:23.042004       1 utils.go:81] GRPC request:
I0818 08:25:23.042042       1 identityserver.go:33] Using default GetPluginInfo
I0818 08:25:23.042050       1 utils.go:86] GRPC response: name:"cinder.csi.openstack.org" vendor_version:"1.2.2@v1.20.0"
I0818 08:25:24.586247       1 utils.go:80] GRPC call: /csi.v1.Node/NodeGetInfo
I0818 08:25:24.586320       1 utils.go:81] GRPC request:
I0818 08:25:24.778203       1 metadata.go:187] Attempting to fetch metadata from http://169.254.169.254/openstack/latest/meta_data.json
I0818 08:25:25.334565       1 utils.go:86] GRPC response: node_id:"2f579c81-5bd6-4eba-89de-3527d980963f" max_volumes_per_node:256 accessible_topology:<segments:<key:"topology.cinder.csi.openstack.org/zone" value:"nova" > >

kubelet of controller:

Aug 18 09:04:56 control01-onap-master-service-mesh kubelet[26953]: I0818 09:04:56.102536   26953 setters.go:86] Using node IP: "10.253.0.91"
Aug 18 09:04:56 control01-onap-master-service-mesh kubelet[26953]: W0818 09:04:56.102574   26953 setters.go:194] adding overridden hostname of control01-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:05:06 control01-onap-master-service-mesh kubelet[26953]: I0818 09:05:06.180663   26953 setters.go:86] Using node IP: "10.253.0.91"
Aug 18 09:05:06 control01-onap-master-service-mesh kubelet[26953]: W0818 09:05:06.180717   26953 setters.go:194] adding overridden hostname of control01-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:05:16 control01-onap-master-service-mesh kubelet[26953]: I0818 09:05:16.271646   26953 setters.go:86] Using node IP: "10.253.0.91"
Aug 18 09:05:16 control01-onap-master-service-mesh kubelet[26953]: W0818 09:05:16.271742   26953 setters.go:194] adding overridden hostname of control01-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:05:26 control01-onap-master-service-mesh kubelet[26953]: I0818 09:05:26.352212   26953 setters.go:86] Using node IP: "10.253.0.91"
Aug 18 09:05:26 control01-onap-master-service-mesh kubelet[26953]: W0818 09:05:26.352376   26953 setters.go:194] adding overridden hostname of control01-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:05:36 control01-onap-master-service-mesh kubelet[26953]: I0818 09:05:36.427269   26953 setters.go:86] Using node IP: "10.253.0.91"
Aug 18 09:05:36 control01-onap-master-service-mesh kubelet[26953]: W0818 09:05:36.427297   26953 setters.go:194] adding overridden hostname of control01-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:05:44 control01-onap-master-service-mesh kubelet[26953]: I0818 09:05:44.494522   26953 kubelet_getters.go:176] "Pod status updated" pod="kube-system/kube-controller-manager-control01-onap-master-service-mesh" status=Running
Aug 18 09:05:44 control01-onap-master-service-mesh kubelet[26953]: I0818 09:05:44.494625   26953 kubelet_getters.go:176] "Pod status updated" pod="kube-system/kube-apiserver-control01-onap-master-service-mesh" status=Running
Aug 18 09:05:44 control01-onap-master-service-mesh kubelet[26953]: I0818 09:05:44.494662   26953 kubelet_getters.go:176] "Pod status updated" pod="kube-system/kube-scheduler-control01-onap-master-service-mesh" status=Running
Aug 18 09:05:46 control01-onap-master-service-mesh kubelet[26953]: I0818 09:05:46.501038   26953 setters.go:86] Using node IP: "10.253.0.91"
Aug 18 09:05:46 control01-onap-master-service-mesh kubelet[26953]: W0818 09:05:46.501086   26953 setters.go:194] adding overridden hostname of control01-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:05:56 control01-onap-master-service-mesh kubelet[26953]: I0818 09:05:56.612754   26953 setters.go:86] Using node IP: "10.253.0.91"
Aug 18 09:05:56 control01-onap-master-service-mesh kubelet[26953]: W0818 09:05:56.613561   26953 setters.go:194] adding overridden hostname of control01-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:06:05 control01-onap-master-service-mesh etcd[23069]: 2021-08-18 09:06:05.224226 I | mvcc: store.index: compact 32541
Aug 18 09:06:05 control01-onap-master-service-mesh etcd[23069]: 2021-08-18 09:06:05.343185 I | mvcc: finished scheduled compaction at 32541 (took 114.653222ms)
Aug 18 09:06:06 control01-onap-master-service-mesh kubelet[26953]: I0818 09:06:06.719655   26953 setters.go:86] Using node IP: "10.253.0.91"
Aug 18 09:06:06 control01-onap-master-service-mesh kubelet[26953]: W0818 09:06:06.719732   26953 setters.go:194] adding overridden hostname of control01-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:06:16 control01-onap-master-service-mesh kubelet[26953]: I0818 09:06:16.792372   26953 setters.go:86] Using node IP: "10.253.0.91"
Aug 18 09:06:16 control01-onap-master-service-mesh kubelet[26953]: W0818 09:06:16.792430   26953 setters.go:194] adding overridden hostname of control01-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:06:26 control01-onap-master-service-mesh kubelet[26953]: I0818 09:06:26.877417   26953 setters.go:86] Using node IP: "10.253.0.91"
Aug 18 09:06:26 control01-onap-master-service-mesh kubelet[26953]: W0818 09:06:26.877459   26953 setters.go:194] adding overridden hostname of control01-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:06:36 control01-onap-master-service-mesh kubelet[26953]: I0818 09:06:36.974543   26953 setters.go:86] Using node IP: "10.253.0.91"
Aug 18 09:06:36 control01-onap-master-service-mesh kubelet[26953]: W0818 09:06:36.974682   26953 setters.go:194] adding overridden hostname of control01-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:06:44 control01-onap-master-service-mesh kubelet[26953]: I0818 09:06:44.495030   26953 kubelet_getters.go:176] "Pod status updated" pod="kube-system/kube-apiserver-control01-onap-master-service-mesh" status=Running
Aug 18 09:06:44 control01-onap-master-service-mesh kubelet[26953]: I0818 09:06:44.495109   26953 kubelet_getters.go:176] "Pod status updated" pod="kube-system/kube-scheduler-control01-onap-master-service-mesh" status=Running
Aug 18 09:06:44 control01-onap-master-service-mesh kubelet[26953]: I0818 09:06:44.495135   26953 kubelet_getters.go:176] "Pod status updated" pod="kube-system/kube-controller-manager-control01-onap-master-service-mesh" status=Running
Aug 18 09:06:47 control01-onap-master-service-mesh kubelet[26953]: I0818 09:06:47.059731   26953 setters.go:86] Using node IP: "10.253.0.91"
Aug 18 09:06:47 control01-onap-master-service-mesh kubelet[26953]: W0818 09:06:47.059783   26953 setters.go:194] adding overridden hostname of control01-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:06:57 control01-onap-master-service-mesh kubelet[26953]: I0818 09:06:57.140711   26953 setters.go:86] Using node IP: "10.253.0.91"
Aug 18 09:06:57 control01-onap-master-service-mesh kubelet[26953]: W0818 09:06:57.140769   26953 setters.go:194] adding overridden hostname of control01-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:07:07 control01-onap-master-service-mesh kubelet[26953]: I0818 09:07:07.238594   26953 setters.go:86] Using node IP: "10.253.0.91"
Aug 18 09:07:07 control01-onap-master-service-mesh kubelet[26953]: W0818 09:07:07.238640   26953 setters.go:194] adding overridden hostname of control01-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:07:17 control01-onap-master-service-mesh kubelet[26953]: I0818 09:07:17.309480   26953 setters.go:86] Using node IP: "10.253.0.91"
Aug 18 09:07:17 control01-onap-master-service-mesh kubelet[26953]: W0818 09:07:17.309516   26953 setters.go:194] adding overridden hostname of control01-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:07:27 control01-onap-master-service-mesh kubelet[26953]: I0818 09:07:27.398142   26953 setters.go:86] Using node IP: "10.253.0.91"
Aug 18 09:07:27 control01-onap-master-service-mesh kubelet[26953]: W0818 09:07:27.398196   26953 setters.go:194] adding overridden hostname of control01-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:07:37 control01-onap-master-service-mesh kubelet[26953]: I0818 09:07:37.461895   26953 setters.go:86] Using node IP: "10.253.0.91"
Aug 18 09:07:37 control01-onap-master-service-mesh kubelet[26953]: W0818 09:07:37.461931   26953 setters.go:194] adding overridden hostname of control01-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:07:44 control01-onap-master-service-mesh kubelet[26953]: I0818 09:07:44.495466   26953 kubelet_getters.go:176] "Pod status updated" pod="kube-system/kube-apiserver-control01-onap-master-service-mesh" status=Running
Aug 18 09:07:44 control01-onap-master-service-mesh kubelet[26953]: I0818 09:07:44.495525   26953 kubelet_getters.go:176] "Pod status updated" pod="kube-system/kube-scheduler-control01-onap-master-service-mesh" status=Running
Aug 18 09:07:44 control01-onap-master-service-mesh kubelet[26953]: I0818 09:07:44.495549   26953 kubelet_getters.go:176] "Pod status updated" pod="kube-system/kube-controller-manager-control01-onap-master-service-mesh" status=Running
Aug 18 09:07:44 control01-onap-master-service-mesh kubelet[26953]: I0818 09:07:44.645510   26953 container_manager_linux.go:490] [ContainerManager]: Discovered runtime cgroups name: /systemd/system.slice
Aug 18 09:07:47 control01-onap-master-service-mesh kubelet[26953]: I0818 09:07:47.548327   26953 setters.go:86] Using node IP: "10.253.0.91"
Aug 18 09:07:47 control01-onap-master-service-mesh kubelet[26953]: W0818 09:07:47.548381   26953 setters.go:194] adding overridden hostname of control01-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:07:57 control01-onap-master-service-mesh kubelet[26953]: I0818 09:07:57.653219   26953 setters.go:86] Using node IP: "10.253.0.91"
Aug 18 09:07:57 control01-onap-master-service-mesh kubelet[26953]: W0818 09:07:57.653506   26953 setters.go:194] adding overridden hostname of control01-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:08:07 control01-onap-master-service-mesh kubelet[26953]: I0818 09:08:07.745887   26953 setters.go:86] Using node IP: "10.253.0.91"
Aug 18 09:08:07 control01-onap-master-service-mesh kubelet[26953]: W0818 09:08:07.745918   26953 setters.go:194] adding overridden hostname of control01-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:08:17 control01-onap-master-service-mesh kubelet[26953]: I0818 09:08:17.817718   26953 setters.go:86] Using node IP: "10.253.0.91"
Aug 18 09:08:17 control01-onap-master-service-mesh kubelet[26953]: W0818 09:08:17.817748   26953 setters.go:194] adding overridden hostname of control01-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:08:27 control01-onap-master-service-mesh kubelet[26953]: I0818 09:08:27.880118   26953 setters.go:86] Using node IP: "10.253.0.91"
Aug 18 09:08:27 control01-onap-master-service-mesh kubelet[26953]: W0818 09:08:27.880156   26953 setters.go:194] adding overridden hostname of control01-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:08:37 control01-onap-master-service-mesh kubelet[26953]: I0818 09:08:37.953261   26953 setters.go:86] Using node IP: "10.253.0.91"
Aug 18 09:08:37 control01-onap-master-service-mesh kubelet[26953]: W0818 09:08:37.953336   26953 setters.go:194] adding overridden hostname of control01-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:08:44 control01-onap-master-service-mesh kubelet[26953]: I0818 09:08:44.495765   26953 kubelet_getters.go:176] "Pod status updated" pod="kube-system/kube-apiserver-control01-onap-master-service-mesh" status=Running
Aug 18 09:08:44 control01-onap-master-service-mesh kubelet[26953]: I0818 09:08:44.495814   26953 kubelet_getters.go:176] "Pod status updated" pod="kube-system/kube-scheduler-control01-onap-master-service-mesh" status=Running
Aug 18 09:08:44 control01-onap-master-service-mesh kubelet[26953]: I0818 09:08:44.495827   26953 kubelet_getters.go:176] "Pod status updated" pod="kube-system/kube-controller-manager-control01-onap-master-service-mesh" status=Running
Aug 18 09:08:48 control01-onap-master-service-mesh kubelet[26953]: I0818 09:08:48.051236   26953 setters.go:86] Using node IP: "10.253.0.91"
Aug 18 09:08:48 control01-onap-master-service-mesh kubelet[26953]: W0818 09:08:48.051285   26953 setters.go:194] adding overridden hostname of control01-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:08:58 control01-onap-master-service-mesh kubelet[26953]: I0818 09:08:58.125128   26953 setters.go:86] Using node IP: "10.253.0.91"
Aug 18 09:08:58 control01-onap-master-service-mesh kubelet[26953]: W0818 09:08:58.125173   26953 setters.go:194] adding overridden hostname of control01-onap-master-service-mesh to cloudprovider-reported addresses

kubeler of used node:

Aug 18 09:05:02 compute06-onap-master-service-mesh kubelet[26532]: I0818 09:05:02.767379   26532 setters.go:86] Using node IP: "10.253.0.9"
Aug 18 09:05:02 compute06-onap-master-service-mesh kubelet[26532]: W0818 09:05:02.767430   26532 setters.go:194] adding overridden hostname of compute06-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:05:03 compute06-onap-master-service-mesh kubelet[26532]: I0818 09:05:03.105225   26532 kubelet.go:1914] SyncLoop (ADD, "api"): "pysimple_default(8b48fed4-adaf-4cfe-89fd-c6bf2c66984b)"
Aug 18 09:05:03 compute06-onap-master-service-mesh kubelet[26532]: I0818 09:05:03.105315   26532 topology_manager.go:187] [topologymanager] Topology Admit Handler
Aug 18 09:05:03 compute06-onap-master-service-mesh kubelet[26532]: I0818 09:05:03.105452   26532 reflector.go:219] Starting reflector *v1.Secret (0s) from object-"default"/"default-token-qtpdx"
Aug 18 09:05:03 compute06-onap-master-service-mesh systemd[1]: Created slice libcontainer container kubepods-besteffort-pod8b48fed4_adaf_4cfe_89fd_c6bf2c66984b.slice.
Aug 18 09:05:03 compute06-onap-master-service-mesh kubelet[26532]: I0818 09:05:03.294104   26532 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-qtpdx" (UniqueName: "kubernetes.io/secret/8b48fed4-adaf-4cfe-89fd-c6bf2c66984b-default-token-qtpdx") pod "pysimple" (UID: "8b48fed4-adaf-4cfe-89fd-c6bf2c66984b")
Aug 18 09:05:03 compute06-onap-master-service-mesh kubelet[26532]: I0818 09:05:03.394883   26532 reconciler.go:269] operationExecutor.MountVolume started for volume "default-token-qtpdx" (UniqueName: "kubernetes.io/secret/8b48fed4-adaf-4cfe-89fd-c6bf2c66984b-default-token-qtpdx") pod "pysimple" (UID: "8b48fed4-adaf-4cfe-89fd-c6bf2c66984b")
Aug 18 09:05:03 compute06-onap-master-service-mesh kubelet[26532]: I0818 09:05:03.404526   26532 operation_generator.go:672] MountVolume.SetUp succeeded for volume "default-token-qtpdx" (UniqueName: "kubernetes.io/secret/8b48fed4-adaf-4cfe-89fd-c6bf2c66984b-default-token-qtpdx") pod "pysimple" (UID: "8b48fed4-adaf-4cfe-89fd-c6bf2c66984b")
Aug 18 09:05:12 compute06-onap-master-service-mesh kubelet[26532]: I0818 09:05:12.861991   26532 setters.go:86] Using node IP: "10.253.0.9"
Aug 18 09:05:12 compute06-onap-master-service-mesh kubelet[26532]: W0818 09:05:12.862057   26532 setters.go:194] adding overridden hostname of compute06-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:05:22 compute06-onap-master-service-mesh kubelet[26532]: I0818 09:05:22.949915   26532 setters.go:86] Using node IP: "10.253.0.9"
Aug 18 09:05:22 compute06-onap-master-service-mesh kubelet[26532]: W0818 09:05:22.951323   26532 setters.go:194] adding overridden hostname of compute06-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:05:33 compute06-onap-master-service-mesh kubelet[26532]: I0818 09:05:33.052712   26532 setters.go:86] Using node IP: "10.253.0.9"
Aug 18 09:05:33 compute06-onap-master-service-mesh kubelet[26532]: W0818 09:05:33.052771   26532 setters.go:194] adding overridden hostname of compute06-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:05:43 compute06-onap-master-service-mesh kubelet[26532]: I0818 09:05:43.142274   26532 setters.go:86] Using node IP: "10.253.0.9"
Aug 18 09:05:43 compute06-onap-master-service-mesh kubelet[26532]: W0818 09:05:43.142337   26532 setters.go:194] adding overridden hostname of compute06-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:05:48 compute06-onap-master-service-mesh kubelet[26532]: I0818 09:05:48.906609   26532 kubelet_getters.go:176] "Pod status updated" pod="kube-system/nginx-proxy-compute06-onap-master-service-mesh" status=Running
Aug 18 09:05:49 compute06-onap-master-service-mesh kubelet[26532]: I0818 09:05:49.068814   26532 container_manager_linux.go:490] [ContainerManager]: Discovered runtime cgroups name: /systemd/system.slice
Aug 18 09:05:53 compute06-onap-master-service-mesh kubelet[26532]: I0818 09:05:53.233621   26532 setters.go:86] Using node IP: "10.253.0.9"
Aug 18 09:05:53 compute06-onap-master-service-mesh kubelet[26532]: W0818 09:05:53.233667   26532 setters.go:194] adding overridden hostname of compute06-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:06:03 compute06-onap-master-service-mesh kubelet[26532]: I0818 09:06:03.323849   26532 setters.go:86] Using node IP: "10.253.0.9"
Aug 18 09:06:03 compute06-onap-master-service-mesh kubelet[26532]: W0818 09:06:03.323906   26532 setters.go:194] adding overridden hostname of compute06-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:06:13 compute06-onap-master-service-mesh kubelet[26532]: I0818 09:06:13.405812   26532 setters.go:86] Using node IP: "10.253.0.9"
Aug 18 09:06:13 compute06-onap-master-service-mesh kubelet[26532]: W0818 09:06:13.405852   26532 setters.go:194] adding overridden hostname of compute06-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:06:23 compute06-onap-master-service-mesh kubelet[26532]: I0818 09:06:23.497428   26532 setters.go:86] Using node IP: "10.253.0.9"
Aug 18 09:06:23 compute06-onap-master-service-mesh kubelet[26532]: W0818 09:06:23.497492   26532 setters.go:194] adding overridden hostname of compute06-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:06:33 compute06-onap-master-service-mesh kubelet[26532]: I0818 09:06:33.594709   26532 setters.go:86] Using node IP: "10.253.0.9"
Aug 18 09:06:33 compute06-onap-master-service-mesh kubelet[26532]: W0818 09:06:33.594785   26532 setters.go:194] adding overridden hostname of compute06-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:06:43 compute06-onap-master-service-mesh kubelet[26532]: I0818 09:06:43.701219   26532 setters.go:86] Using node IP: "10.253.0.9"
Aug 18 09:06:43 compute06-onap-master-service-mesh kubelet[26532]: W0818 09:06:43.701262   26532 setters.go:194] adding overridden hostname of compute06-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:06:48 compute06-onap-master-service-mesh kubelet[26532]: I0818 09:06:48.906908   26532 kubelet_getters.go:176] "Pod status updated" pod="kube-system/nginx-proxy-compute06-onap-master-service-mesh" status=Running
Aug 18 09:06:53 compute06-onap-master-service-mesh kubelet[26532]: I0818 09:06:53.790450   26532 setters.go:86] Using node IP: "10.253.0.9"
Aug 18 09:06:53 compute06-onap-master-service-mesh kubelet[26532]: W0818 09:06:53.790524   26532 setters.go:194] adding overridden hostname of compute06-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:07:03 compute06-onap-master-service-mesh kubelet[26532]: I0818 09:07:03.898645   26532 setters.go:86] Using node IP: "10.253.0.9"
Aug 18 09:07:03 compute06-onap-master-service-mesh kubelet[26532]: W0818 09:07:03.898737   26532 setters.go:194] adding overridden hostname of compute06-onap-master-service-mesh to cloudprovider-reported addresses
Aug 18 09:07:06 compute06-onap-master-service-mesh kubelet[26532]: E0818 09:07:06.117585   26532 kubelet.go:1679] Unable to attach or mount volumes for pod "pysimple_default(8b48fed4-adaf-4cfe-89fd-c6bf2c66984b)": unmounted volumes=[data-volume], unattached volumes=[data-volume default-token-qtpdx]: timed out waiting for the condition; skipping pod
Aug 18 09:07:06 compute06-onap-master-service-mesh kubelet[26532]: E0818 09:07:06.117679   26532 pod_workers.go:191] Error syncing pod 8b48fed4-adaf-4cfe-89fd-c6bf2c66984b ("pysimple_default(8b48fed4-adaf-4cfe-89fd-c6bf2c66984b)"), skipping: unmounted volumes=[data-volume], unattached volumes=[data-volume default-token-qtpdx]: timed out waiting for the condition
Aug 18 09:07:14 compute06-onap-master-service-mesh kubelet[26532]: I0818 09:07:14.001385   26532 setters.go:86] Using node IP: "10.253.0.9"
Aug 18 09:07:14 compute06-onap-master-service-mesh kubelet[26532]: W0818 09:07:14.001448   26532 setters.go:194] adding overridden hostname of compute06-onap-master-service-mesh to cloudprovider-reported addresses

sylvainOL commented 3 years ago

I've also taken a look at changes in kubespray and I don't see anything that might be an issue these previous weeks: https://github.com/kubernetes-sigs/kubespray/commits/release-2.16

Another change is that we've update debian base image to a more recent one (issue with stable change to oldstable) but I'm not sure it changes something :(

sylvainOL commented 3 years ago

oh and on another deployment, (k8s v1.19.9 deployed with kubespray release-2.15), but using the in tree provisioner (AFAIK, I don't see csi pods), it has the same issue and there I can see:

Aug 18 10:26:16 compute09-onap-master kubelet[23979]: I0818 10:26:16.208002   23979 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-9a5f3728-71a6-476c-b314-c5597c4b3c7b" (UniqueName: "kubernetes.io/cinder/72ebe3ae-6c00-4938-a846-641d6db41424") pod "pysimple" (UID: "eae15b76-2dc6-4b45-a8f8-37afef64d215")
Aug 18 10:26:16 compute09-onap-master kubelet[23979]: I0818 10:26:16.208127   23979 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "default-token-c5mn9" (UniqueName: "kubernetes.io/secret/eae15b76-2dc6-4b45-a8f8-37afef64d215-default-token-c5mn9") pod "pysimple" (UID: "eae15b76-2dc6-4b45-a8f8-37afef64d215")
Aug 18 10:26:16 compute09-onap-master kubelet[23979]: E0818 10:26:16.208381   23979 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/cinder/72ebe3ae-6c00-4938-a846-641d6db41424 podName: nodeName:}" failed. No retries permitted until 2021-08-18 10:26:16.708289302 +0000 UTC m=+36961.681443996 (durationBeforeRetry 500ms). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-9a5f3728-71a6-476c-b314-c5597c4b3c7b\" (UniqueName: \"kubernetes.io/cinder/72ebe3ae-6c00-4938-a846-641d6db41424\") pod \"pysimple\" (UID: \"eae15b76-2dc6-4b45-a8f8-37afef64d215\") "
Aug 18 10:26:16 compute09-onap-master kubelet[23979]: I0818 10:26:16.309176   23979 reconciler.go:269] operationExecutor.MountVolume started for volume "default-token-c5mn9" (UniqueName: "kubernetes.io/secret/eae15b76-2dc6-4b45-a8f8-37afef64d215-default-token-c5mn9") pod "pysimple" (UID: "eae15b76-2dc6-4b45-a8f8-37afef64d215")
Aug 18 10:26:16 compute09-onap-master systemd[1]: Started Kubernetes transient mount for /var/lib/kubelet/pods/eae15b76-2dc6-4b45-a8f8-37afef64d215/volumes/kubernetes.io~secret/default-token-c5mn9.
Aug 18 10:26:16 compute09-onap-master kubelet[23979]: I0818 10:26:16.326420   23979 operation_generator.go:663] MountVolume.SetUp succeeded for volume "default-token-c5mn9" (UniqueName: "kubernetes.io/secret/eae15b76-2dc6-4b45-a8f8-37afef64d215-default-token-c5mn9") pod "pysimple" (UID: "eae15b76-2dc6-4b45-a8f8-37afef64d215")
Aug 18 10:26:16 compute09-onap-master kubelet[23979]: I0818 10:26:16.712822   23979 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-9a5f3728-71a6-476c-b314-c5597c4b3c7b" (UniqueName: "kubernetes.io/cinder/72ebe3ae-6c00-4938-a846-641d6db41424") pod "pysimple" (UID: "eae15b76-2dc6-4b45-a8f8-37afef64d215")
Aug 18 10:26:16 compute09-onap-master kubelet[23979]: E0818 10:26:16.713403   23979 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/cinder/72ebe3ae-6c00-4938-a846-641d6db41424 podName: nodeName:}" failed. No retries permitted until 2021-08-18 10:26:17.713179577 +0000 UTC m=+36962.686334383 (durationBeforeRetry 1s). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-9a5f3728-71a6-476c-b314-c5597c4b3c7b\" (UniqueName: \"kubernetes.io/cinder/72ebe3ae-6c00-4938-a846-641d6db41424\") pod \"pysimple\" (UID: \"eae15b76-2dc6-4b45-a8f8-37afef64d215\") "
Aug 18 10:26:17 compute09-onap-master kubelet[23979]: I0818 10:26:17.720696   23979 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-9a5f3728-71a6-476c-b314-c5597c4b3c7b" (UniqueName: "kubernetes.io/cinder/72ebe3ae-6c00-4938-a846-641d6db41424") pod "pysimple" (UID: "eae15b76-2dc6-4b45-a8f8-37afef64d215")
Aug 18 10:26:17 compute09-onap-master kubelet[23979]: E0818 10:26:17.720928   23979 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/cinder/72ebe3ae-6c00-4938-a846-641d6db41424 podName: nodeName:}" failed. No retries permitted until 2021-08-18 10:26:19.720850651 +0000 UTC m=+36964.694005342 (durationBeforeRetry 2s). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-9a5f3728-71a6-476c-b314-c5597c4b3c7b\" (UniqueName: \"kubernetes.io/cinder/72ebe3ae-6c00-4938-a846-641d6db41424\") pod \"pysimple\" (UID: \"eae15b76-2dc6-4b45-a8f8-37afef64d215\") "
Aug 18 10:26:17 compute09-onap-master kubelet[23979]: I0818 10:26:17.867484   23979 kubelet_getters.go:178] "Pod status updated" pod="kube-system/nginx-proxy-compute09-onap-master" status=Running
Aug 18 10:26:19 compute09-onap-master kubelet[23979]: I0818 10:26:19.694219   23979 setters.go:77] Using node IP: "10.253.0.172"
Aug 18 10:26:19 compute09-onap-master kubelet[23979]: W0818 10:26:19.694320   23979 setters.go:179] adding overridden hostname of compute09-onap-master to cloudprovider-reported addresses
Aug 18 10:26:19 compute09-onap-master kubelet[23979]: I0818 10:26:19.733278   23979 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-9a5f3728-71a6-476c-b314-c5597c4b3c7b" (UniqueName: "kubernetes.io/cinder/72ebe3ae-6c00-4938-a846-641d6db41424") pod "pysimple" (UID: "eae15b76-2dc6-4b45-a8f8-37afef64d215")
Aug 18 10:26:19 compute09-onap-master kubelet[23979]: E0818 10:26:19.733545   23979 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/cinder/72ebe3ae-6c00-4938-a846-641d6db41424 podName: nodeName:}" failed. No retries permitted until 2021-08-18 10:26:23.733452437 +0000 UTC m=+36968.706607187 (durationBeforeRetry 4s). Error: "Volume has not been added to the list of VolumesInUse in the node's volume status for volume \"pvc-9a5f3728-71a6-476c-b314-c5597c4b3c7b\" (UniqueName: \"kubernetes.io/cinder/72ebe3ae-6c00-4938-a846-641d6db41424\") pod \"pysimple\" (UID: \"eae15b76-2dc6-4b45-a8f8-37afef64d215\") "
Aug 18 10:26:23 compute09-onap-master kubelet[23979]: I0818 10:26:23.760904   23979 reconciler.go:224] operationExecutor.VerifyControllerAttachedVolume started for volume "pvc-9a5f3728-71a6-476c-b314-c5597c4b3c7b" (UniqueName: "kubernetes.io/cinder/72ebe3ae-6c00-4938-a846-641d6db41424") pod "pysimple" (UID: "eae15b76-2dc6-4b45-a8f8-37afef64d215")
Aug 18 10:26:23 compute09-onap-master kubelet[23979]: I0818 10:26:23.768255   23979 operation_generator.go:1348] Controller attach succeeded for volume "pvc-9a5f3728-71a6-476c-b314-c5597c4b3c7b" (UniqueName: "kubernetes.io/cinder/72ebe3ae-6c00-4938-a846-641d6db41424") pod "pysimple" (UID: "eae15b76-2dc6-4b45-a8f8-37afef64d215") device path: "/dev/sdf"
Aug 18 10:26:23 compute09-onap-master kubelet[23979]: I0818 10:26:23.861547   23979 reconciler.go:269] operationExecutor.MountVolume started for volume "pvc-9a5f3728-71a6-476c-b314-c5597c4b3c7b" (UniqueName: "kubernetes.io/cinder/72ebe3ae-6c00-4938-a846-641d6db41424") pod "pysimple" (UID: "eae15b76-2dc6-4b45-a8f8-37afef64d215")
Aug 18 10:26:23 compute09-onap-master kubelet[23979]: I0818 10:26:23.861768   23979 operation_generator.go:558] MountVolume.WaitForAttach entering for volume "pvc-9a5f3728-71a6-476c-b314-c5597c4b3c7b" (UniqueName: "kubernetes.io/cinder/72ebe3ae-6c00-4938-a846-641d6db41424") pod "pysimple" (UID: "eae15b76-2dc6-4b45-a8f8-37afef64d215") DevicePath "/dev/sdf"
Aug 18 10:26:25 compute09-onap-master kubelet[23979]: W0818 10:26:25.570278   23979 openstack_volumes.go:576] Failed to find device for the volumeID: "72ebe3ae-6c00-4938-a846-641d6db41424"
Aug 18 10:26:25 compute09-onap-master kubelet[23979]: E0818 10:26:25.570353   23979 attacher.go:249] Error: could not find attached Cinder disk "72ebe3ae-6c00-4938-a846-641d6db41424" (path: ""): <nil>
Aug 18 10:26:28 compute09-onap-master kubelet[23979]: W0818 10:26:28.099386   23979 openstack_volumes.go:576] Failed to find device for the volumeID: "72ebe3ae-6c00-4938-a846-641d6db41424"
Aug 18 10:26:28 compute09-onap-master kubelet[23979]: E0818 10:26:28.099425   23979 attacher.go:249] Error: could not find attached Cinder disk "72ebe3ae-6c00-4938-a846-641d6db41424" (path: ""): <nil>
Aug 18 10:26:29 compute09-onap-master kubelet[23979]: I0818 10:26:29.787654   23979 setters.go:77] Using node IP: "10.253.0.172"
Aug 18 10:26:29 compute09-onap-master kubelet[23979]: W0818 10:26:29.789542   23979 setters.go:179] adding overridden hostname of compute09-onap-master to cloudprovider-reported addresses
Aug 18 10:26:32 compute09-onap-master kubelet[23979]: W0818 10:26:32.621000   23979 openstack_volumes.go:576] Failed to find device for the volumeID: "72ebe3ae-6c00-4938-a846-641d6db41424"
Aug 18 10:26:32 compute09-onap-master kubelet[23979]: E0818 10:26:32.621069   23979 attacher.go:249] Error: could not find attached Cinder disk "72ebe3ae-6c00-4938-a846-641d6db41424" (path: ""): <nil>
Aug 18 10:26:39 compute09-onap-master kubelet[23979]: I0818 10:26:39.886535   23979 setters.go:77] Using node IP: "10.253.0.172"
Aug 18 10:26:39 compute09-onap-master kubelet[23979]: W0818 10:26:39.886590   23979 setters.go:179] adding overridden hostname of compute09-onap-master to cloudprovider-reported addresses
Aug 18 10:26:41 compute09-onap-master kubelet[23979]: W0818 10:26:41.338150   23979 openstack_volumes.go:576] Failed to find device for the volumeID: "72ebe3ae-6c00-4938-a846-641d6db41424"
Aug 18 10:26:41 compute09-onap-master kubelet[23979]: E0818 10:26:41.338234   23979 attacher.go:249] Error: could not find attached Cinder disk "72ebe3ae-6c00-4938-a846-641d6db41424" (path: ""): <nil>
Aug 18 10:26:49 compute09-onap-master kubelet[23979]: I0818 10:26:49.988059   23979 setters.go:77] Using node IP: "10.253.0.172"
Aug 18 10:26:49 compute09-onap-master kubelet[23979]: W0818 10:26:49.988098   23979 setters.go:179] adding overridden hostname of compute09-onap-master to cloudprovider-reported addresses
Aug 18 10:26:58 compute09-onap-master kubelet[23979]: W0818 10:26:58.082561   23979 openstack_volumes.go:576] Failed to find device for the volumeID: "72ebe3ae-6c00-4938-a846-641d6db41424"
Aug 18 10:26:58 compute09-onap-master kubelet[23979]: E0818 10:26:58.082631   23979 attacher.go:249] Error: could not find attached Cinder disk "72ebe3ae-6c00-4938-a846-641d6db41424" (path: ""): <nil>
Aug 18 10:27:00 compute09-onap-master kubelet[23979]: I0818 10:27:00.068286   23979 setters.go:77] Using node IP: "10.253.0.172"
Aug 18 10:27:00 compute09-onap-master kubelet[23979]: W0818 10:27:00.068342   23979 setters.go:179] adding overridden hostname of compute09-onap-master to cloudprovider-reported addresses
Aug 18 10:27:10 compute09-onap-master kubelet[23979]: I0818 10:27:10.141330   23979 setters.go:77] Using node IP: "10.253.0.172"
Aug 18 10:27:10 compute09-onap-master kubelet[23979]: W0818 10:27:10.141382   23979 setters.go:179] adding overridden hostname of compute09-onap-master to cloudprovider-reported addresses
Aug 18 10:27:17 compute09-onap-master kubelet[23979]: I0818 10:27:17.868071   23979 kubelet_getters.go:178] "Pod status updated" pod="kube-system/nginx-proxy-compute09-onap-master" status=Running
Aug 18 10:27:20 compute09-onap-master kubelet[23979]: I0818 10:27:20.248321   23979 setters.go:77] Using node IP: "10.253.0.172"
Aug 18 10:27:20 compute09-onap-master kubelet[23979]: W0818 10:27:20.248389   23979 setters.go:179] adding overridden hostname of compute09-onap-master to cloudprovider-reported addresses
Aug 18 10:27:25 compute09-onap-master kubelet[23979]: W0818 10:27:25.408096   23979 openstack_volumes.go:576] Failed to find device for the volumeID: "2dcde027-dc08-477a-a269-0764538c163b"
Aug 18 10:27:25 compute09-onap-master kubelet[23979]: E0818 10:27:25.408164   23979 attacher.go:249] Error: could not find attached Cinder disk "2dcde027-dc08-477a-a269-0764538c163b" (path: ""): <nil>
Aug 18 10:27:30 compute09-onap-master kubelet[23979]: I0818 10:27:30.341445   23979 setters.go:77] Using node IP: "10.253.0.172"
Aug 18 10:27:30 compute09-onap-master kubelet[23979]: W0818 10:27:30.342138   23979 setters.go:179] adding overridden hostname of compute09-onap-master to cloudprovider-reported addresses
Aug 18 10:27:30 compute09-onap-master kubelet[23979]: W0818 10:27:30.590876   23979 openstack_volumes.go:576] Failed to find device for the volumeID: "72ebe3ae-6c00-4938-a846-641d6db41424"
Aug 18 10:27:30 compute09-onap-master kubelet[23979]: E0818 10:27:30.590907   23979 attacher.go:249] Error: could not find attached Cinder disk "72ebe3ae-6c00-4938-a846-641d6db41424" (path: ""): <nil>
Aug 18 10:27:34 compute09-onap-master kubelet[23979]: E0818 10:27:34.579596   23979 kubelet.go:1625] Unable to attach or mount volumes for pod "pysimple_default(0ac1ce61-3853-428d-8247-284e992f6ab4)": unmounted volumes=[data-volume default-token-c5mn9], unattached volumes=[data-volume default-token-c5mn9]: timed out waiting for the condition; skipping pod
Aug 18 10:27:34 compute09-onap-master kubelet[23979]: E0818 10:27:34.579679   23979 pod_workers.go:191] Error syncing pod 0ac1ce61-3853-428d-8247-284e992f6ab4 ("pysimple_default(0ac1ce61-3853-428d-8247-284e992f6ab4)"), skipping: unmounted volumes=[data-volume default-token-c5mn9], unattached volumes=[data-volume default-token-c5mn9]: timed out waiting for the condition

sylvainOL commented 3 years ago

continuing to find the issue, I've taken a look at an "old" deployment (using the old debian version but also being deployed via victoria but running now on wallaby) same version as previous one (k8s v1.19.9 deployed with kubespray release-2.15) and here it works!

the difference is on the naming of disk in /dev/disk/by-id/:

old one:

ls -l /dev/disk/by-id/
total 0
lrwxrwxrwx 1 root root  9 Aug 18 12:16 virtio-20841276-a7f7-4558-9 -> ../../vdc
lrwxrwxrwx 1 root root 10 Aug 18 12:16 virtio-20841276-a7f7-4558-9-part1 -> ../../vdc1
lrwxrwxrwx 1 root root  9 Aug 18 12:16 virtio-47f49d38-a2a2-4a29-8 -> ../../vdb
lrwxrwxrwx 1 root root 10 Aug 18 12:16 virtio-47f49d38-a2a2-4a29-8-part1 -> ../../vdb1
lrwxrwxrwx 1 root root  9 Aug 18 12:16 virtio-84411c8d-7df1-45af-9 -> ../../vde
lrwxrwxrwx 1 root root  9 Aug 18 12:16 virtio-c9a59f31-bb54-4821-a -> ../../vdd
lrwxrwxrwx 1 root root 10 Aug 18 12:16 virtio-c9a59f31-bb54-4821-a-part1 -> ../../vdd1

new one:

ls -l /dev/disk/by-id/
total 0
lrwxrwxrwx 1 root root  9 Aug 18 12:02 scsi-0QEMU_QEMU_HARDDISK_4443ca08-531f-4719-b5d6-41c293c43d8b -> ../../sdc
lrwxrwxrwx 1 root root 10 Aug 18 12:02 scsi-0QEMU_QEMU_HARDDISK_4443ca08-531f-4719-b5d6-41c293c43d8b-part1 -> ../../sdc1
lrwxrwxrwx 1 root root  9 Aug 18 12:02 scsi-0QEMU_QEMU_HARDDISK_5630acac-2a21-4afb-b720-006ef7ed5d68 -> ../../sdb
lrwxrwxrwx 1 root root 10 Aug 18 12:02 scsi-0QEMU_QEMU_HARDDISK_5630acac-2a21-4afb-b720-006ef7ed5d68-part1 -> ../../sdb1
lrwxrwxrwx 1 root root  9 Aug 18 12:02 scsi-0QEMU_QEMU_HARDDISK_72ebe3ae-6c00-4938-a846-641d6db41424 -> ../../sdf
lrwxrwxrwx 1 root root  9 Aug 18 12:02 scsi-0QEMU_QEMU_HARDDISK_cda1af67-579d-40ef-8aab-173308c4004d -> ../../sdd
lrwxrwxrwx 1 root root 10 Aug 18 12:02 scsi-0QEMU_QEMU_HARDDISK_cda1af67-579d-40ef-8aab-173308c4004d-part1 -> ../../sdd1
lrwxrwxrwx 1 root root  9 Aug 18 12:02 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0 -> ../../sda
lrwxrwxrwx 1 root root 10 Aug 18 12:02 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0-part1 -> ../../sda1
lrwxrwxrwx 1 root root 11 Aug 18 12:02 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0-part14 -> ../../sda14
lrwxrwxrwx 1 root root 11 Aug 18 12:02 scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0-part15 -> ../../sda15

but when I look at https://github.com/kubernetes/cloud-provider-openstack/blob/release-1.20/pkg/util/mount/mount.go#L158, it should be good no?

sylvainOL commented 3 years ago

Ok, so first sorry to speak out loud here but I may have found something on the first issue: I'm deploying openstack csi but using kubelet provided openstack cloud adapter...

so I'm able to see the issue in kubelet from node:

Aug 18 12:47:24 compute06-onap-master-service-mesh kubelet[26532]: W0818 12:47:24.047291   26532 openstack_volumes.go:576] Failed to find device for the volumeID: "485f7181-8fa5-458d-8f24-73bde6e698ff"
Aug 18 12:47:24 compute06-onap-master-service-mesh kubelet[26532]: E0818 12:47:24.047352   26532 attacher.go:249] Error: could not find attached Cinder disk "485f7181-8fa5-458d-8f24-73bde6e698ff" (path: ""): <nil>

So I guess I'm not using cinder csi but the "old" way in kubelet.

And with the new version in libvirt, it's broken now.

So I'll close this one, sorry for the noise!

lingxiankong commented 3 years ago

All good @sylvainOL, please let us know if you have any further questions.

kubernetes / cloud-provider-openstack

[cinder-csi-plugin] Unable to attach or mount volumes: timed out waiting for the condition #1628