Volumes go missing. - Githubissues

davidkarlsen commented 3 years ago

What steps did you take and what happened: Created a pvc, which got formatted and mounted just fine. Then booted the machine, after bootup the volume is not to be found! This causes loss of data and is very severe.

What did you expect to happen: Not loosing volumes!

The output of the following commands will help us better understand what's going on: (Pasting long output into a GitHub gist or other Pastebin is fine.)

kubectl logs -f openebs-lvm-controller-0 -n kube-system -c openebs-lvm-plugin


I0919 21:07:44.486525       1 main.go:136] LVM Driver Version :- 0.8.0 - commit :- 929ae4439f2da71a2d6ee5bda6a33dd2f7d424fc
I0919 21:07:44.486615       1 main.go:137] DriverName: local.csi.openebs.io Plugin: controller EndPoint: unix:///var/lib/csi/sockets/pluginproxy/csi.sock NodeID:  SetIOLimits: false ContainerRuntime: containerd RIopsPerGB: [] WIopsPerGB: [] RBpsPerGB: [] WBpsPerGB: []
I0919 21:07:44.486646       1 driver.go:48] enabling volume access mode: SINGLE_NODE_WRITER
I0919 21:07:44.496097       1 controller.go:212] waiting for k8s & lvm node informer caches to be synced
I0919 21:07:44.596266       1 controller.go:216] synced k8s & lvm node informer caches
I0919 21:07:44.596300       1 controller.go:218] initializing csi provisioning leak protection controller
I0919 21:07:44.596895       1 leak_protection.go:129] "starting up csi pvc controller"
I0919 21:07:44.596916       1 shared_informer.go:240] Waiting for caches to sync for CSI Provisioner
I0919 21:07:44.597275       1 grpc.go:190] Listening for connections on address: &net.UnixAddr{Name:"//var/lib/csi/sockets/pluginproxy/csi.sock", Net:"unix"}
I0919 21:07:44.697073       1 shared_informer.go:247] Caches are synced for CSI Provisioner 
I0919 21:07:44.991733       1 grpc.go:72] GRPC call: /csi.v1.Identity/Probe requests {}
I0919 21:07:44.999709       1 grpc.go:81] GRPC response: {}
I0919 21:07:45.001288       1 grpc.go:72] GRPC call: /csi.v1.Identity/GetPluginInfo requests {}
I0919 21:07:45.001331       1 grpc.go:81] GRPC response: {"name":"local.csi.openebs.io","vendor_version":"0.8.0"}
I0919 21:07:45.002064       1 grpc.go:72] GRPC call: /csi.v1.Identity/GetPluginCapabilities requests {}
I0919 21:07:45.002142       1 grpc.go:81] GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"Service":{"type":2}}}]}
I0919 21:07:45.002973       1 grpc.go:72] GRPC call: /csi.v1.Controller/ControllerGetCapabilities requests {}
I0919 21:07:45.003009       1 grpc.go:81] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":4}}}]}
I0919 21:07:45.325452       1 grpc.go:72] GRPC call: /csi.v1.Identity/Probe requests {}
I0919 21:07:45.325666       1 grpc.go:81] GRPC response: {}
I0919 21:07:45.326637       1 grpc.go:72] GRPC call: /csi.v1.Identity/GetPluginInfo requests {}
I0919 21:07:45.326779       1 grpc.go:81] GRPC response: {"name":"local.csi.openebs.io","vendor_version":"0.8.0"}
I0919 21:07:45.327324       1 grpc.go:72] GRPC call: /csi.v1.Identity/GetPluginCapabilities requests {}
I0919 21:07:45.327354       1 grpc.go:81] GRPC response: {"capabilities":[{"Type":{"Service":{"type":1}}},{"Type":{"Service":{"type":2}}}]}
I0919 21:07:45.328054       1 grpc.go:72] GRPC call: /csi.v1.Controller/ControllerGetCapabilities requests {}
I0919 21:07:45.328135       1 grpc.go:81] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":4}}}]}
I0919 21:07:45.673459       1 grpc.go:72] GRPC call: /csi.v1.Identity/GetPluginInfo requests {}
I0919 21:07:45.673695       1 grpc.go:81] GRPC response: {"name":"local.csi.openebs.io","vendor_version":"0.8.0"}
I0919 21:07:45.675001       1 grpc.go:72] GRPC call: /csi.v1.Identity/Probe requests {}
I0919 21:07:45.675122       1 grpc.go:81] GRPC response: {}
I0919 21:07:45.675962       1 grpc.go:72] GRPC call: /csi.v1.Controller/ControllerGetCapabilities requests {}
I0919 21:07:45.676000       1 grpc.go:81] GRPC response: {"capabilities":[{"Type":{"Rpc":{"type":1}}},{"Type":{"Rpc":{"type":9}}},{"Type":{"Rpc":{"type":5}}},{"Type":{"Rpc":{"type":4}}}]}
....
I0919 22:20:15.437512       1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-ksx-g-c01oco06","openebs.io/nodename":"alt-ksx-g-c01oco06"}},"parameters":{"fsType":"xfs","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.437667       1 grpc.go:81] GRPC response: {"available_capacity":261124784128}
I0919 22:20:15.438064       1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-eos-g-c01oco03","openebs.io/nodename":"alt-eos-g-c01oco03"}},"parameters":{"fsType":"xfs","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.438195       1 grpc.go:81] GRPC response: {"available_capacity":322118352896}
I0919 22:20:15.438548       1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-ebs-g-c01oco01","openebs.io/nodename":"alt-ebs-g-c01oco01"}},"parameters":{"fsType":"ext4","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.438674       1 grpc.go:81] GRPC response: {"available_capacity":322118352896}
I0919 22:20:15.439082       1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-ksx-g-c01oco09","openebs.io/nodename":"alt-ksx-g-c01oco09"}},"parameters":{"fsType":"ext4","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.439263       1 grpc.go:81] GRPC response: {"available_capacity":2171097579520}
I0919 22:20:15.439687       1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-eos-g-c01oco02","openebs.io/nodename":"alt-eos-g-c01oco02"}},"parameters":{"fsType":"xfs","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.439817       1 grpc.go:81] GRPC response: {"available_capacity":322118352896}
I0919 22:20:15.440257       1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-ksx-g-c01oco06","openebs.io/nodename":"alt-ksx-g-c01oco06"}},"parameters":{"fsType":"ext4","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.440385       1 grpc.go:81] GRPC response: {"available_capacity":261124784128}
I0919 22:20:15.440940       1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-ksx-g-c01oco04","openebs.io/nodename":"alt-ksx-g-c01oco04"}},"parameters":{"fsType":"xfs","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.441111       1 grpc.go:81] GRPC response: {"available_capacity":261124784128}
I0919 22:20:15.441526       1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-ksx-g-c01oco07","openebs.io/nodename":"alt-ksx-g-c01oco07"}},"parameters":{"fsType":"ext4","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.441646       1 grpc.go:81] GRPC response: {"available_capacity":2171097579520}
I0919 22:20:15.441994       1 grpc.go:72] GRPC call: /csi.v1.Controller/GetCapacity requests {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-ebs-g-c01oco01","openebs.io/nodename":"alt-ebs-g-c01oco01"}},"parameters":{"fsType":"xfs","shared":"yes","thinProvision":"no","volgroup":"datavg"},"volume_capabilities":[{"AccessType":{"Mount":{}},"access_mode":{}}]}
I0919 22:20:15.442129       1 grpc.go:81] GRPC response: {"available_capacity":322118352896}


* `kubectl logs -f openebs-lvm-node-[xxxx] -n kube-system -c openebs-lvm-plugin`
* `kubectl get pods -n kube-system`

k -n openebs get pods NAME READY STATUS RESTARTS AGE openebs-admission-server-894884484-gcc5f 1/1 Running 0 29h openebs-apiserver-6494d57b96-xfltt 1/1 Running 0 77m openebs-localpv-provisioner-5787758b59-tzs8l 1/1 Running 1 29h openebs-lvm-localpv-controller-0 5/5 Running 0 76m openebs-lvm-localpv-node-7x2kw 2/2 Running 6 5d4h openebs-lvm-localpv-node-98kxr 2/2 Running 6 5d4h openebs-lvm-localpv-node-9r2kd 2/2 Running 20 5d4h openebs-lvm-localpv-node-h949v 2/2 Running 6 5d4h openebs-lvm-localpv-node-jff5w 2/2 Running 6 5d4h openebs-lvm-localpv-node-kccx5 2/2 Running 6 5d4h openebs-lvm-localpv-node-ln7r4 2/2 Running 6 5d4h openebs-lvm-localpv-node-pzxw4 2/2 Running 6 5d4h openebs-lvm-localpv-node-qxwn9 2/2 Running 11 5d4h openebs-lvm-localpv-node-qzxzb 2/2 Running 6 5d4h openebs-lvm-localpv-node-trfvk 2/2 Running 6 5d4h openebs-lvm-localpv-node-w25kv 2/2 Running 6 5d4h openebs-lvm-localpv-node-wxv6w 2/2 Running 6 5d4h openebs-lvm-localpv-node-xkmzj 2/2 Running 6 5d4h openebs-ndm-5rq88 1/1 Running 5 10d openebs-ndm-ftnwd 1/1 Running 3 10d openebs-ndm-k9z5w 1/1 Running 3 10d openebs-ndm-mxzss 1/1 Running 3 10d openebs-ndm-operator-7bd7455b5b-khrmt 1/1 Running 0 29h openebs-ndm-p79zz 1/1 Running 3 10d openebs-ndm-p7f77 1/1 Running 3 10d openebs-ndm-phfjw 1/1 Running 3 10d openebs-ndm-zsqmp 1/1 Running 3 10d openebs-provisioner-9f948db4-ljndj 1/1 Running 1 29h openebs-snapshot-operator-68cf9946d6-vc5ls 2/2 Running 0 77m

I0919 21:47:33.032487 I0919 21:47:33.032679 I0919 21:47:33.032725 I0919 21:47:33.033388 I0919 21:47:33.036307 I0919 21:47:33.036946 I0919 21:47:33.037042 I0919 21:47:33.037049 I0919 21:47:33.036950 I0919 21:47:33.037584 I0919 21:47:33.055970 I0919 21:47:33.056056 I0919 21:47:33.067860 I0919 21:47:33.068028 I0919 21:47:33.068466 I0919 21:47:33.071142 I0919 21:47:33.137317 I0919 21:47:33.137350 I0919 21:47:33.137728 I0919 21:47:33.137738 I0919 21:47:33.137830 I0919 21:47:33.137840 I0919 21:47:33.137832 I0919 21:47:33.137904 I0919 21:47:33.168649 I0919 21:47:33.168667 I0919 21:47:33.489540 I0919 21:47:33.489636 I0919 21:47:33.497190 I0919 21:47:33.497209 I0919 21:47:33.754135 I0919 21:47:33.755194 I0919 21:47:35.327404 I0919 21:47:35.334632 I0919 21:47:36.692782 I0919 21:47:36.762805 E0919 21:47:36.927823 The file /dev/datavg/ ) E0919 21:47:36.927901 The file /dev/datavg/ ) E0919 21:47:36.927997 The file /dev/datavg/ ) I0919 21:47:37.493233 I0919 21:47:37.498794 E0919 21:47:37.499905 The file /dev/datavg/ ) E0919 21:47:37.499975 The file /dev/datavg/ ) E0919 21:47:37.500016 The file /dev/datavg/ ) I0919 21:47:37.818527 I0919 21:47:37.823301 E0919 21:47:37.824462 The file /dev/datavg/ ) E0919 21:47:37.824478 The file /dev/datavg/ ) E0919 21:47:37.824495 The file /dev/datavg/ ) I0919 21:47:38.411354 I0919 21:47:38.417422 E0919 21:47:38.418586 The file /dev/datavg/ ) E0919 21:47:38.418630 The file /dev/datavg/ ) E0919 21:47:38.418687 The file /dev/datavg/ ) I0919 21:47:38.520481 I0919 21:47:38.525817 E0919 21:47:38.526895 The file /dev/datavg/ ) ... I0919 21:48:33.198169 I0919 21:48:33.198235 I0919 21:48:33.205156 I0919 21:48:33.205174 I0919 21:48:40.776987 I0919 21:48:40.793493 E0919 21:48:40.794577 The file /dev/datavg/ ) E0919 21:48:40.794632 The file /dev/datavg/ ) E0919 21:48:40.794757 The file /dev/datavg/ ) I0919 21:48:41.690835 I0919 21:48:41.697249 E0919 21:48:41.698425 The file /dev/datavg/ ) E0919 21:48:41.698442 The file /dev/datavg/ ) E0919 21:48:41.698463 The file /dev/datavg/ ) 1 main.go:136] LVM Driver Version :- 0.8.0 - commit :- 929ae4439f2da71a2d6ee5bda6a33dd2f7d424fc 1 main.go:137] DriverName: local.csi.openebs.io Plugin: agent EndPoint: unix:///plugin/csi.sock NodeID: alt-ksx-g-c01oco03 SetIOLimits: false ContainerRuntime: containerd RIopsPerGB: [] WIopsPerGB: [] RBpsPerGB: [] WBpsPerGB: [] 1 driver.go:48] enabling volume access mode: SINGLE_NODE_WRITER 1 grpc.go:190] Listening for connections on address: &net.UnixAddr{Name:"//plugin/csi.sock", Net:"unix"} 1 builder.go:107] Creating event broadcaster 1 builder.go:107] Creating event broadcaster 1 snapshot.go:165] Starting Snap controller 1 snapshot.go:168] Waiting for informer caches to sync 1 volume.go:265] Starting Vol controller 1 volume.go:268] Waiting for informer caches to sync 1 volume.go:209] Got add event for Vol pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca 1 volume.go:209] Got add event for Vol pvc-63ebce02-7077-48ea-88b3-301b1cf7909d 1 builder.go:116] Creating event broadcaster 1 lvmnode.go:191] Starting Node controller 1 lvmnode.go:194] Waiting for informer caches to sync 1 lvmnode.go:127] Got add event for lvm node openebs/alt-ksx-g-c01oco03 1 snapshot.go:172] Starting Snap workers 1 snapshot.go:179] Started Snap workers 1 volume.go:272] Starting Vol workers 1 volume.go:279] Started Vol workers 1 volume.go:98] lvm volume already provisioned 1 volume.go:337] Successfully synced 'openebs/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca' 1 volume.go:98] lvm volume already provisioned 1 volume.go:337] Successfully synced 'openebs/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d' 1 lvmnode.go:199] Starting Node workers 1 lvmnode.go:206] Started Node workers 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}] 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp: DeletionGracePeriodSeconds: Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc00026a060 BlockOwnerDeletion:}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]} 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03' 1 grpc.go:72] GRPC call: /csi.v1.Identity/GetPluginInfo requests {} 1 grpc.go:81] GRPC response: {"name":"local.csi.openebs.io","vendor_version":"0.8.0"} 1 grpc.go:72] GRPC call: /csi.v1.Node/NodeGetInfo requests {} 1 grpc.go:81] GRPC response: {"accessible_topology":{"segments":{"kubernetes.io/hostname":"alt-ksx-g-c01oco03","openebs.io/nodename":"alt-ksx-g-c01oco03"}},"node_id":"alt-ksx-g-c01oco03"} 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"} 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca] 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"} 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca] 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"} 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d] 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"} 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d] 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"} 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca] 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}] 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp: DeletionGracePeriodSeconds: Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0006224c9 BlockOwnerDeletion:}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]} 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03' 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"} 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca] 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"} 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d] 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified.

I0919 21:58:33.193143 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}] I0919 21:58:33.193387 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp: DeletionGracePeriodSeconds: Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc00026bce8 BlockOwnerDeletion:}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]} I0919 21:58:33.200591 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03 I0919 21:58:33.200709 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03' I0919 21:59:33.197599 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}] I0919 21:59:33.197854 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp: DeletionGracePeriodSeconds: Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0006320f9 BlockOwnerDeletion:}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]} I0919 21:59:33.203510 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03 I0919 21:59:33.203600 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03' I0919 21:59:55.071045 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"} I0919 21:59:55.077098 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca] E0919 21:59:55.078420 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) E0919 21:59:55.078595 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) E0919 21:59:55.078709 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) I0919 21:59:56.077218 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"} I0919 21:59:56.081863 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d] E0919 21:59:56.083050 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) E0919 21:59:56.083154 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) E0919 21:59:56.083251 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) I0919 22:00:33.194333 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}] I0919 22:00:33.194546 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp: DeletionGracePeriodSeconds: Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000131069 BlockOwnerDeletion:}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]} I0919 22:00:33.200632 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03 I0919 22:00:33.200795 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03' I0919 22:01:33.192396 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}] I0919 22:01:33.192458 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp: DeletionGracePeriodSeconds: Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0006a808a BlockOwnerDeletion:}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]} I0919 22:01:33.199002 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03 I0919 22:01:33.199019 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03' I0919 22:01:57.130951 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"} I0919 22:01:57.136043 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca] E0919 22:01:57.137124 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) E0919 22:01:57.137250 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) E0919 22:01:57.137358 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) I0919 22:01:58.139894 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"} I0919 22:01:58.145250 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d] E0919 22:01:58.146369 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) E0919 22:01:58.146436 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) E0919 22:01:58.146499 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) I0919 22:02:33.198541 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}] I0919 22:02:33.198810 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp: DeletionGracePeriodSeconds: Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000632248 BlockOwnerDeletion:}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]} I0919 22:02:33.219623 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03 I0919 22:02:33.219775 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03' I0919 22:03:33.197462 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}] I0919 22:03:33.197666 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp: DeletionGracePeriodSeconds: Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0006221bb BlockOwnerDeletion:}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]} I0919 22:03:33.204957 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03 I0919 22:03:33.205147 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03' I0919 22:03:59.214157 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"} I0919 22:03:59.221739 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca] E0919 22:03:59.223076 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) E0919 22:03:59.223095 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) E0919 22:03:59.223124 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) I0919 22:04:00.225863 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"} I0919 22:04:00.230920 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d] E0919 22:04:00.232305 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) E0919 22:04:00.232440 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) E0919 22:04:00.232540 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) I0919 22:04:33.193373 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}] I0919 22:04:33.193540 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp: DeletionGracePeriodSeconds: Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000043278 BlockOwnerDeletion:}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]} I0919 22:04:33.199806 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03 I0919 22:04:33.199958 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03' I0919 22:05:33.201598 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}] I0919 22:05:33.201968 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp: DeletionGracePeriodSeconds: Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0003e00aa BlockOwnerDeletion:}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]} I0919 22:05:33.209690 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03 I0919 22:05:33.209786 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03' I0919 22:06:01.309472 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"} I0919 22:06:01.315687 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca] E0919 22:06:01.316691 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) E0919 22:06:01.316707 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) E0919 22:06:01.316734 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) I0919 22:06:02.317247 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"} I0919 22:06:02.321985 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d] E0919 22:06:02.323009 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) E0919 22:06:02.323099 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) E0919 22:06:02.323181 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) I0919 22:06:33.192108 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}] I0919 22:06:33.192174 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp: DeletionGracePeriodSeconds: Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000212090 BlockOwnerDeletion:}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]} I0919 22:06:33.203863 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03 I0919 22:06:33.203885 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03' I0919 22:07:33.197599 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}] I0919 22:07:33.197883 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp: DeletionGracePeriodSeconds: Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000212000 BlockOwnerDeletion:}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]} I0919 22:07:33.204772 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03 I0919 22:07:33.204789 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03' I0919 22:08:03.336415 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"} I0919 22:08:03.343673 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca] E0919 22:08:03.344630 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) E0919 22:08:03.344645 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) E0919 22:08:03.344675 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) I0919 22:08:04.353996 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"} I0919 22:08:04.364752 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d] E0919 22:08:04.365823 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) E0919 22:08:04.365942 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) E0919 22:08:04.366035 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) I0919 22:08:33.195376 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}] I0919 22:08:33.195672 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp: DeletionGracePeriodSeconds: Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0006ac7e8 BlockOwnerDeletion:}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]} I0919 22:08:33.205158 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03 I0919 22:08:33.205327 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03' I0919 22:09:33.195347 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}] I0919 22:09:33.195593 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp: DeletionGracePeriodSeconds: Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0005901da BlockOwnerDeletion:}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]} I0919 22:09:33.204502 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03 I0919 22:09:33.204669 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03' I0919 22:10:05.398785 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"} I0919 22:10:05.405224 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca] E0919 22:10:05.406422 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) E0919 22:10:05.406541 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) E0919 22:10:05.406634 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) I0919 22:10:06.407071 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"} I0919 22:10:06.412408 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d] E0919 22:10:06.413575 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) E0919 22:10:06.413705 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) E0919 22:10:06.413795 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) I0919 22:10:33.195388 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}] I0919 22:10:33.195460 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp: DeletionGracePeriodSeconds: Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000622978 BlockOwnerDeletion:}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]} I0919 22:10:33.209769 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03 I0919 22:10:33.209910 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03' I0919 22:11:33.195346 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}] I0919 22:11:33.195445 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp: DeletionGracePeriodSeconds: Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0000421f0 BlockOwnerDeletion:}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]} I0919 22:11:33.206672 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03 I0919 22:11:33.206754 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03' I0919 22:12:07.452479 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"} I0919 22:12:07.457948 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca] E0919 22:12:07.459041 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) E0919 22:12:07.459053 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) E0919 22:12:07.459073 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) I0919 22:12:08.468111 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"} I0919 22:12:08.473579 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d] E0919 22:12:08.474930 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) E0919 22:12:08.475057 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) E0919 22:12:08.475153 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) I0919 22:12:33.199283 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}] I0919 22:12:33.199554 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp: DeletionGracePeriodSeconds: Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0001ab668 BlockOwnerDeletion:}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]} I0919 22:12:33.208417 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03 I0919 22:12:33.208537 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03' I0919 22:13:33.194398 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}] I0919 22:13:33.194604 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp: DeletionGracePeriodSeconds: Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0001aa000 BlockOwnerDeletion:}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]} I0919 22:13:33.202748 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03 I0919 22:13:33.202825 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03' I0919 22:14:09.561523 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"} I0919 22:14:09.566213 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca] E0919 22:14:09.567230 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) E0919 22:14:09.567328 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) E0919 22:14:09.567438 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) I0919 22:14:10.571841 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"} I0919 22:14:10.577746 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d] E0919 22:14:10.579041 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) E0919 22:14:10.579115 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) E0919 22:14:10.579204 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) I0919 22:14:33.194217 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}] I0919 22:14:33.194488 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp: DeletionGracePeriodSeconds: Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0001300e9 BlockOwnerDeletion:}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]} I0919 22:14:33.203751 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03 I0919 22:14:33.203929 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03' I0919 22:15:33.197368 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s:511996Mi Format:BinarySI} Free:{i:{value:209375461376 scale:0} d:{Dec:} s:199676Mi Format:BinarySI} LVCount:2 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}] I0919 22:15:33.197592 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind:LVMNode APIVersion:local.openebs.io/v1alpha1} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305723725 Generation:4 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp: DeletionGracePeriodSeconds: Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc00026a0c9 BlockOwnerDeletion:}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]} I0919 22:15:33.204602 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03 I0919 22:15:33.204716 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03' I0919 22:15:33.204919 1 lvmnode.go:139] Got update event for lvm node openebs/alt-ksx-g-c01oco03 I0919 22:15:33.227390 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s:511996Mi Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:} s:506876Mi Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}] I0919 22:15:33.227723 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305975699 Generation:5 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp: DeletionGracePeriodSeconds: Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc00026a6f0 BlockOwnerDeletion:}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]} I0919 22:15:33.237650 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03 I0919 22:15:33.237836 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03' I0919 22:16:11.616127 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"} I0919 22:16:11.621182 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca] E0919 22:16:11.622289 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) E0919 22:16:11.622376 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) E0919 22:16:11.622450 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) I0919 22:16:12.629181 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"} I0919 22:16:12.634759 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d] E0919 22:16:12.638315 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) E0919 22:16:12.638330 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) E0919 22:16:12.638354 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) I0919 22:16:33.194321 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s:511996Mi Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:} s:506876Mi Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}] I0919 22:16:33.194582 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305975699 Generation:5 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp: DeletionGracePeriodSeconds: Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0001ab10d BlockOwnerDeletion:}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]} I0919 22:16:33.200940 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03 I0919 22:16:33.201043 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03' I0919 22:17:33.195250 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s:511996Mi Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:} s:506876Mi Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}] I0919 22:17:33.195449 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305975699 Generation:5 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp: DeletionGracePeriodSeconds: Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000800000 BlockOwnerDeletion:}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]} I0919 22:17:33.203085 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03 I0919 22:17:33.203146 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03' I0919 22:18:13.705268 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"} I0919 22:18:13.710264 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca] E0919 22:18:13.711379 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) E0919 22:18:13.711399 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) E0919 22:18:13.711423 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) I0919 22:18:14.712131 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"} I0919 22:18:14.717477 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d] E0919 22:18:14.718574 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) E0919 22:18:14.718671 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) E0919 22:18:14.718739 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) I0919 22:18:33.197552 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s:511996Mi Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:} s:506876Mi Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}] I0919 22:18:33.197616 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305975699 Generation:5 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp: DeletionGracePeriodSeconds: Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000212598 BlockOwnerDeletion:}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]} I0919 22:18:33.203316 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03 I0919 22:18:33.203334 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03' I0919 22:19:33.198358 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s:511996Mi Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:} s:506876Mi Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}] I0919 22:19:33.198425 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305975699 Generation:5 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp: DeletionGracePeriodSeconds: Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000130000 BlockOwnerDeletion:}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]} I0919 22:19:33.204133 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03 I0919 22:19:33.204295 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03' I0919 22:20:15.780801 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"} I0919 22:20:15.785438 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca] E0919 22:20:15.786634 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) E0919 22:20:15.786799 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) E0919 22:20:15.786892 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) I0919 22:20:16.798260 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"} I0919 22:20:16.805115 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d] E0919 22:20:16.806421 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) E0919 22:20:16.806537 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) E0919 22:20:16.806623 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. )

E0919 22:22:17.863627 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca [ext4] to /var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount, error format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) E0919 22:22:17.863684 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) I0919 22:22:18.863589 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"vault-1","csi.storage.k8s.io/pod.namespace":"vault","csi.storage.k8s.io/pod.uid":"160b5143-fb8d-4758-a829-6c92d8c719d7","csi.storage.k8s.io/serviceAccount.name":"vault","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-63ebce02-7077-48ea-88b3-301b1cf7909d"} I0919 22:22:18.869062 1 mount_linux.go:366] Disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d] E0919 22:22:18.870243 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) E0919 22:22:18.870264 1 mount.go:72] lvm: failed to mount volume /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d [ext4] to /var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount, error format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) E0919 22:22:18.870295 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d" failed: type:("ext4") target:("/var/lib/kubelet/pods/160b5143-fb8d-4758-a829-6c92d8c719d7/volumes/kubernetes.io~csi/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-63ebce02-7077-48ea-88b3-301b1cf7909d does not exist and no size was specified. ) I0919 22:22:33.195255 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s:511996Mi Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:} s:506876Mi Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}] I0919 22:22:33.195326 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305975699 Generation:5 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp: DeletionGracePeriodSeconds: Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0006ac5ea BlockOwnerDeletion:}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]} I0919 22:22:33.201762 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03 I0919 22:22:33.201935 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03' I0919 22:23:33.197380 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s:511996Mi Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:} s:506876Mi Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:0 MetadataUsedCount:0 MetadataFree:{i:{value:0 scale:0} d:{Dec:} s: Format:} MetadataSize:{i:{value:0 scale:0} d:{Dec:} s: Format:} Permission:0 AllocationPolicy:0}], required=[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}] I0919 22:23:33.197443 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305975699 Generation:5 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp: DeletionGracePeriodSeconds: Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc0000421f8 BlockOwnerDeletion:}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]} I0919 22:23:33.204502 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03 I0919 22:23:33.204519 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03' I0919 22:24:19.937883 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"alertmanager-main-2","csi.storage.k8s.io/pod.namespace":"openshift-monitoring","csi.storage.k8s.io/pod.uid":"2a1d2646-68b5-4916-99b3-18d3918d4831","csi.storage.k8s.io/serviceAccount.name":"alertmanager-main","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"datavg","storage.kubernetes.io/csiProvisionerIdentity":"1631990632881-8081-local.csi.openebs.io"},"volume_id":"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca"} I0919 22:24:19.942809 1 mount_linux.go:366] Disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca] E0919 22:24:19.943870 1 mount_linux.go:372] format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020) The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified. ) ... I0919 22:25:33.199426 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:alt-ksx-g-c01oco03 GenerateName: Namespace:openebs SelfLink: UID:e14c6a98-a813-4250-b3a4-b9362444f91d ResourceVersion:1305975699 Generation:5 CreationTimestamp:2021-08-23 12:07:57 +0000 UTC DeletionTimestamp: DeletionGracePeriodSeconds: Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:alt-ksx-g-c01oco03 UID:c61ff898-2ecb-401c-9c1d-63f9acbbe57f Controller:0xc000130079 BlockOwnerDeletion:}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2021-08-23 12:07:57 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"c61ff898-2ecb-401c-9c1d-63f9acbbe57f\"}":{".":{},"f:apiVersion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:datavg UUID:52aZzd-VsHc-gsWF-TXf6-vEFx-gXdZ-pbho0p Size:{i:{value:536866717696 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:531498008576 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:1 PVCount:1 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:1 MetadataUsedCount:1 MetadataFree:{i:{value:520192 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0} {Name:rootvg UUID:Z1mRPp-5Goy-2wuF-XQbo-uALK-YeyF-EGU6aR Size:{i:{value:267890196480 scale:0} d:{Dec:} s: Format:BinarySI} Free:{i:{value:86855647232 scale:0} d:{Dec:} s: Format:BinarySI} LVCount:5 PVCount:2 MaxLV:0 MaxPV:0 SnapCount:0 MissingPVCount:0 MetadataCount:2 MetadataUsedCount:2 MetadataFree:{i:{value:518656 scale:0} d:{Dec:} s: Format:BinarySI} MetadataSize:{i:{value:1044480 scale:0} d:{Dec:} s: Format:BinarySI} Permission:0 AllocationPolicy:0}]} I0919 22:25:33.220225 1 lvmnode.go:114] lvm node controller: updated node object openebs/alt-ksx-g-c01oco03 I0919 22:25:33.220325 1 lvmnode.go:274] Successfully synced 'openebs/alt-ksx-g-c01oco03'

* `kubectl get lvmvol -A -o yaml`

apiVersion: v1 items:

apiVersion: local.openebs.io/v1alpha1 kind: LVMVolume metadata: creationTimestamp: "2021-09-19T16:58:40Z" finalizers:
- lvm.openebs.io/finalizer generation: 3 labels: kubernetes.io/nodename: alt-ksx-g-c01oco03 name: pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca namespace: openebs resourceVersion: "1304748995" uid: 303e29b0-39f7-413a-9a1a-cecd32757061 spec: capacity: "1073741824" ownerNodeID: alt-ksx-g-c01oco03 shared: "yes" thinProvision: "no" vgPattern: ^datavg$ volGroup: datavg status: state: Ready
apiVersion: local.openebs.io/v1alpha1 kind: LVMVolume metadata: creationTimestamp: "2021-09-19T17:48:47Z" finalizers:
- lvm.openebs.io/finalizer generation: 3 labels: kubernetes.io/nodename: alt-ksx-g-c01oco03 name: pvc-63ebce02-7077-48ea-88b3-301b1cf7909d namespace: openebs resourceVersion: "1304943732" uid: 79c320fc-4a6f-4c0e-b02e-0d433d9cf74f spec: capacity: "1073741824" ownerNodeID: alt-ksx-g-c01oco03 shared: "yes" thinProvision: "no" vgPattern: ^datavg$ volGroup: datavg status: state: Ready

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

 pvscan --cache
[root@alt-ksx-g-c01oco03 ~]# lvscan 
  ACTIVE            '/dev/datavg/openebs' [5.00 GiB] inherit
  ACTIVE            '/dev/rootvg/swap_lv' [2.00 GiB] inherit
  ACTIVE            '/dev/rootvg/tmp_lv' [<8.79 GiB] inherit
  ACTIVE            '/dev/rootvg/var_lv' [150.00 GiB] inherit
  ACTIVE            '/dev/rootvg/home_lv' [1000.00 MiB] inherit
  ACTIVE            '/dev/rootvg/root_lv' [<6.84 GiB] inherit

k get pvc alertmanager-main-db-alertmanager-main-2  -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: local.csi.openebs.io
    volume.kubernetes.io/selected-node: alt-ksx-g-c01oco03
  creationTimestamp: "2021-09-19T16:58:40Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    alertmanager: main
    app: alertmanager
    app.kubernetes.io/instance: main
    app.kubernetes.io/managed-by: prometheus-operator
    app.kubernetes.io/name: alertmanager
    app.kubernetes.io/version: 0.21.0
  name: alertmanager-main-db-alertmanager-main-2
  namespace: openshift-monitoring
  resourceVersion: "1304749087"
  uid: 630461fc-7dbb-4db1-a9fb-0b6f4035c6ca
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: openebs-lvmpv-ext4
  volumeMode: Filesystem
  volumeName: pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  phase: Bound

k describe pod alertmanager-main-2   |tail -10
  Warning  FailedMount       34m                 kubelet            Unable to attach or mount volumes: unmounted volumes=[alertmanager-main-db], unattached volumes=[alertmanager-trusted-ca-bundle kube-api-access-7drhb config-volume tls-assets alertmanager-main-db secret-alertmanager-main-tls secret-alertmanager-main-proxy secret-alertmanager-kube-rbac-proxy]: timed out waiting for the condition
  Warning  FailedMount       28m                 kubelet            Unable to attach or mount volumes: unmounted volumes=[alertmanager-main-db], unattached volumes=[secret-alertmanager-main-proxy secret-alertmanager-kube-rbac-proxy alertmanager-trusted-ca-bundle kube-api-access-7drhb config-volume tls-assets alertmanager-main-db secret-alertmanager-main-tls]: timed out waiting for the condition
  Warning  FailedMount       23m (x4 over 39m)   kubelet            Unable to attach or mount volumes: unmounted volumes=[alertmanager-main-db], unattached volumes=[secret-alertmanager-kube-rbac-proxy alertmanager-trusted-ca-bundle kube-api-access-7drhb config-volume tls-assets alertmanager-main-db secret-alertmanager-main-tls secret-alertmanager-main-proxy]: timed out waiting for the condition
  Warning  FailedMount       21m                 kubelet            Unable to attach or mount volumes: unmounted volumes=[alertmanager-main-db], unattached volumes=[alertmanager-main-db secret-alertmanager-main-tls secret-alertmanager-main-proxy secret-alertmanager-kube-rbac-proxy alertmanager-trusted-ca-bundle kube-api-access-7drhb config-volume tls-assets]: timed out waiting for the condition
  Warning  FailedMount       19m                 kubelet            Unable to attach or mount volumes: unmounted volumes=[alertmanager-main-db], unattached volumes=[tls-assets alertmanager-main-db secret-alertmanager-main-tls secret-alertmanager-main-proxy secret-alertmanager-kube-rbac-proxy alertmanager-trusted-ca-bundle kube-api-access-7drhb config-volume]: timed out waiting for the condition
  Warning  FailedMount       10m (x23 over 41m)  kubelet            MountVolume.SetUp failed for volume "pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" : rpc error: code = Internal desc = failed to format and mount the volume error: format of disk "/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" failed: type:("ext4") target:("/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)
The file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.
)
  Warning  FailedMount  5m35s              kubelet  Unable to attach or mount volumes: unmounted volumes=[alertmanager-main-db], unattached volumes=[kube-api-access-7drhb config-volume tls-assets alertmanager-main-db secret-alertmanager-main-tls secret-alertmanager-main-proxy secret-alertmanager-kube-rbac-proxy alertmanager-trusted-ca-bundle]: timed out waiting for the condition
  Warning  FailedMount  62s (x2 over 30m)  kubelet  Unable to attach or mount volumes: unmounted volumes=[alertmanager-main-db], unattached volumes=[config-volume tls-assets alertmanager-main-db secret-alertmanager-main-tls secret-alertmanager-main-proxy secret-alertmanager-kube-rbac-proxy alertmanager-trusted-ca-bundle kube-api-access-7drhb]: timed out waiting for the condition

k get pv|grep pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca
pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca   1Gi        RWO            Delete           Bound    openshift-monitoring/alertmanager-main-db-alertmanager-main-2                               openebs-lvmpv-ext4            5h30m

k get pv  pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca -o yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: local.csi.openebs.io
  creationTimestamp: "2021-09-19T16:58:41Z"
  finalizers:
  - kubernetes.io/pv-protection
  name: pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca
  resourceVersion: "1304749080"
  uid: c1f1c9dd-6440-459a-9e08-213e68657e45
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: alertmanager-main-db-alertmanager-main-2
    namespace: openshift-monitoring
    resourceVersion: "1304748971"
    uid: 630461fc-7dbb-4db1-a9fb-0b6f4035c6ca
  csi:
    driver: local.csi.openebs.io
    fsType: ext4
    volumeAttributes:
      openebs.io/cas-type: localpv-lvm
      openebs.io/volgroup: datavg
      storage.kubernetes.io/csiProvisionerIdentity: 1631990632881-8081-local.csi.openebs.io
    volumeHandle: pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: openebs.io/nodename
          operator: In
          values:
          - alt-ksx-g-c01oco03
  persistentVolumeReclaimPolicy: Delete
  storageClassName: openebs-lvmpv-ext4
  volumeMode: Filesystem
status:
  phase: Bound

oot@alt-ksx-g-c01oco03 ~]# lvm version
  LVM version:     2.02.187(2)-RHEL7 (2020-03-24)
  Library version: 1.02.170-RHEL7 (2020-03-24)
  Driver version:  4.37.1
  Configuration:   ./configure --build=x86_64-redhat-linux-gnu --host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-default-dm-run-dir=/run --with-default-run-dir=/run/lvm --with-default-pid-dir=/run --with-default-locking-dir=/run/lock/lvm --with-usrlibdir=/usr/lib64 --enable-lvm1_fallback --enable-fsadm --with-pool=internal --enable-write_install --with-user= --with-group= --with-device-uid=0 --with-device-gid=6 --with-device-mode=0660 --enable-pkgconfig --enable-applib --enable-cmdlib --enable-dmeventd --enable-blkid_wiping --enable-python2-bindings --with-cluster=internal --with-clvmd=corosync --enable-cmirrord --with-udevdir=/usr/lib/udev/rules.d --enable-udev_sync --with-thin=internal --enable-lvmetad --with-cache=internal --enable-lvmpolld --enable-lvmlockd-dlm --enable-lvmlockd-sanlock --enable-dmfilemapd
[root@alt-ksx-g-c01oco03 ~]#

Environment:

LVM Driver version:
Kubernetes version (use kubectl version): Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1+9807387", GitCommit:"98073871f173baaa04dc2bafab50effd62c308a6", GitTreeState:"clean", BuildDate:"2021-08-06T14:06:43Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}
Kubernetes installer & version: openshift 4.8
Cloud provider or hardware configuration: n/a
OS (e.g. from /etc/os-release): RHEL 7.x

davidkarlsen commented 3 years ago

see in the messages logs how it was fine before boot, then goes missing 40mins later when booting:

lertmanager-main-db") pod "550264f7-6dc2-4365-96f0-7d640a16b85f" (UID: "550264f7-6dc2-4365-96f0-7d640a16b85f"). InnerVolumeSpecName "pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca". PluginName "kubernetes.io/csi", VolumeGidValue ""
Sep 19 23:07:06 alt-ksx-g-c01oco03 hyperkube: I0919 23:07:06.195980    3082 reconciler.go:312] "operationExecutor.UnmountDevice started for volume \"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca\" (UniqueName: \"kubernetes.io/csi/local.csi.openebs.io^pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca\") on node \"alt-ksx-g-c01oco03\" "
Sep 19 23:07:06 alt-ksx-g-c01oco03 hyperkube: I0919 23:07:06.203350    3082 operation_generator.go:930] UnmountDevice succeeded for volume "pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" %!(EXTRA string=UnmountDevice succeeded for volume "pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" (UniqueName: "kubernetes.io/csi/local.csi.openebs.io^pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca") on node "alt-ksx-g-c01oco03" )
Sep 19 23:07:06 alt-ksx-g-c01oco03 hyperkube: I0919 23:07:06.296581    3082 reconciler.go:319] "Volume detached for volume \"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca\" (UniqueName: \"kubernetes.io/csi/local.csi.openebs.io^pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca\") on node \"alt-ksx-g-c01oco03\" DevicePath \"\""
Sep 19 23:47:36 alt-ksx-g-c01oco03 hyperkube: I0919 23:47:36.573075    3065 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca\" (UniqueName: \"kubernetes.io/csi/local.csi.openebs.io^pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca\") pod \"alertmanager-main-2\" (UID: \"2a1d2646-68b5-4916-99b3-18d3918d4831\") "
Sep 19 23:47:36 alt-ksx-g-c01oco03 hyperkube: I0919 23:47:36.677388    3065 reconciler.go:269] "operationExecutor.MountVolume started for volume \"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca\" (UniqueName: \"kubernetes.io/csi/local.csi.openebs.io^pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca\") pod \"alertmanager-main-2\" (UID: \"2a1d2646-68b5-4916-99b3-18d3918d4831\") "
Sep 19 23:47:36 alt-ksx-g-c01oco03 hyperkube: I0919 23:47:36.685679    3065 operation_generator.go:616] MountVolume.MountDevice succeeded for volume "pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca" (UniqueName: "kubernetes.io/csi/local.csi.openebs.io^pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca") pod "alertmanager-main-2" (UID: "2a1d2646-68b5-4916-99b3-18d3918d4831") device mount path "/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/globalmount"
Sep 19 23:47:36 alt-ksx-g-c01oco03 hyperkube: E0919 23:47:36.928568    3065 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/local.csi.openebs.io^pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca podName: nodeName:}" failed. No retries permitted until 2021-09-19 23:47:37.428526511 +0200 CEST m=+32.344727832 (durationBeforeRetry 500ms). Error: "MountVolume.SetUp failed for volume \"pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca\" (UniqueName: \"kubernetes.io/csi/local.csi.openebs.io^pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca\") pod \"alertmanager-main-2\" (UID: \"2a1d2646-68b5-4916-99b3-18d3918d4831\") : rpc error: code = Internal desc = failed to format and mount the volume error: format of disk \"/dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca\" failed: type:(\"ext4\") target:(\"/var/lib/kubelet/pods/2a1d2646-68b5-4916-99b3-18d3918d4831/volumes/kubernetes.io~csi/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca/mount\") options:(\"defaults\") errcode:(exit status 1) output:(mke2fs 1.45.6 (20-Mar-2020)\nThe file /dev/datavg/pvc-630461fc-7dbb-4db1-a9fb-0b6f4035c6ca does not exist and no size was specified.\n) "

I have seen this on several pvc's, so it's not a single isolated case.

helm -n openebs list NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION openebs openebs 35 2021-09-16 23:51:51.718139269 +0000 UTC deployed openebs-2.12.7 2.12.1

kmova commented 3 years ago

@davidkarlsen -- few questions:

What is the size of the volume group?
How many volumes are you creating on the volumes group and their total size, when the above issue starts to show up?
Can you run the lvm commands to see if the volumes really exist on the node and the capacity information of volumes and the corresponding vg.

davidkarlsen commented 3 years ago

around 300G, plenty of space
a few 1-3
I've run pvscan --cache and lvscan, not to be found at all (!)

davidkarlsen commented 3 years ago

this is the SC definition:

k get sc openebs-lvmpv-ext4  -o yaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"allowVolumeExpansion":true,"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"kustomize.toolkit.fluxcd.io/checksum":"ffec0c8cb91220cd2484eb20ea0dffa03a63bb68"},"labels":{"kustomize.toolkit.fluxcd.io/name":"flux-system","kustomize.toolkit.fluxcd.io/namespace":"flux-system"},"name":"openebs-lvmpv-ext4"},"parameters":{"fsType":"ext4","shared":"yes","thinProvision":"no","volgroup":"datavg"},"provisioner":"local.csi.openebs.io","reclaimPolicy":"Delete","volumeBindingMode":"WaitForFirstConsumer"}
    kustomize.toolkit.fluxcd.io/checksum: ffec0c8cb91220cd2484eb20ea0dffa03a63bb68
  creationTimestamp: "2021-09-19T16:42:58Z"
  labels:
    kustomize.toolkit.fluxcd.io/name: flux-system
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: openebs-lvmpv-ext4
  resourceVersion: "1309101171"
  uid: ba5dd4a1-503b-45cd-8d2f-54aadf92753d
parameters:
  fsType: ext4
  shared: "yes"
  thinProvision: "no"
  volgroup: datavg
provisioner: local.csi.openebs.io
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

kmova commented 3 years ago

@davidkarlsen - I wonder if this may be related to https://access.redhat.com/solutions/2175491

pawanpraka1 commented 3 years ago

@davidkarlsen Currently we support ubuntu based systems only, if possible, try to change the host OS to ubuntu. Also, is the issue reproducible everytime you reboot the machine?

davidkarlsen commented 3 years ago

@davidkarlsen Currently we support ubuntu based systems only, if possible, try to change the host OS to ubuntu. Also, is the issue reproducible everytime you reboot the machine?

not every single time, but very often. as this is openshift, we can't use ubuntu

davidkarlsen commented 3 years ago

@davidkarlsen - I wonder if this may be related to access.redhat.com/solutions/2175491

Hmm, how can I figure out if multipath is in use? But I doubt it as we're on:

uname -a
Linux alt-ksx-g-c01oco01 3.10.0-1160.36.2.el7.x86_64 #1 SMP Thu Jul 8 02:53:40 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
-sh-4.2$ cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.9 (Maipo)

which should not be affected.

kmova commented 3 years ago

How is the reboot of the machine triggered? Does it clear all the volumes? Can you try to capture the lvm config file after volumes are provisioned and post the reboot to check if this is actually a config sync issue?

Also we are trying to put together a troubleshooting guide for LVM metadata corruption cases and how to recover from them. Here is the draft version: https://docs.google.com/document/d/1qB7VjPHrCczLlSg9gX6X_3QWiuEOoObXINl9ovMKmjU/edit

(cc: @prateekpandey14 )

davidkarlsen commented 3 years ago

How is the reboot of the machine triggered? Does it clear all the volumes? Can you try to capture the lvm config file after volumes are provisioned and post the reboot to check if this is actually a config sync issue?

No - it's not set to clear all volumes as reboot! :-) we cordon the node, then drain all but daemonsets, with options: --delete-emptydir-data --ignore-daemonsets

indeed I do not find any references to the volumes in the lvm config! (this is prior to any reboot):

lvscan 
  ACTIVE            '/dev/datavg/openebs' [5.00 GiB] inherit
  ACTIVE            '/dev/datavg/pvc-222d5156-3507-4f43-8323-8368a0b114d7' [1.00 GiB] inherit
  ACTIVE            '/dev/datavg/pvc-ef24242b-90ec-4f51-bc52-928e331b86d3' [100.00 GiB] inherit
  ACTIVE            '/dev/datavg/pvc-e997ee44-85f8-4b07-9785-ac9324acb1b5' [100.00 GiB] inherit
  ACTIVE            '/dev/rootvg/swap_lv' [2.00 GiB] inherit
  ACTIVE            '/dev/rootvg/tmp_lv' [<8.79 GiB] inherit
  ACTIVE            '/dev/rootvg/var_lv' [150.00 GiB] inherit
  ACTIVE            '/dev/rootvg/home_lv' [1000.00 MiB] inherit
  ACTIVE            '/dev/rootvg/root_lv' [<6.84 GiB] inherit

find /etc/lvm/ -type f|xargs fgrep -il 'pvc-222d5156-3507-4f43-8323-8368a0b114d7'
[root@alt-ksx-g-c01oco01 lvm]# find /etc/lvm/ -type f|xargs fgrep -il pvc-ef24242b-90ec-4f51-bc52-928e331b86d3
[root@alt-ksx-g-c01oco01 lvm]# find /etc/lvm/ -type f|xargs fgrep -il pvc-e997ee44-85f8-4b07-9785-ac9324acb1b5

davidkarlsen commented 3 years ago

could this perhaps be a se-linux issue? I assume the lvm command are executed into the host-system from one of the openebs containers, and that might not have enough privilegies?

SCCs in openshift: https://docs.openshift.com/container-platform/4.8/authentication/managing-security-context-constraints.html#default-sccs_configuring-internal-oauth

k get rolebinding system:openshift:scc:privileged-openebs -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"rbac.authorization.k8s.io/v1","kind":"RoleBinding","metadata":{"annotations":{"kustomize.toolkit.fluxcd.io/checksum":"ffec0c8cb91220cd2484eb20ea0dffa03a63bb68"},"labels":{"kustomize.toolkit.fluxcd.io/name":"flux-system","kustomize.toolkit.fluxcd.io/namespace":"flux-system"},"name":"system:openshift:scc:privileged-openebs","namespace":"openebs"},"roleRef":{"apiGroup":"rbac.authorization.k8s.io","kind":"ClusterRole","name":"system:openshift:scc:privileged"},"subjects":[{"kind":"ServiceAccount","name":"openebs","namespace":"openebs"},{"kind":"ServiceAccount","name":"openebs-lvm-node-sa","namespace":"openebs"}]}
    kustomize.toolkit.fluxcd.io/checksum: ffec0c8cb91220cd2484eb20ea0dffa03a63bb68
  creationTimestamp: "2020-12-19T13:04:22Z"
  labels:
    kustomize.toolkit.fluxcd.io/name: flux-system
    kustomize.toolkit.fluxcd.io/namespace: flux-system
  name: system:openshift:scc:privileged-openebs
  namespace: openebs
  resourceVersion: "1309101387"
  uid: f1e32386-120c-40a7-b624-5b5188539d34
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:openshift:scc:privileged
subjects:
- kind: ServiceAccount
  name: openebs
  namespace: openebs
- kind: ServiceAccount
  name: openebs-lvm-node-sa
  namespace: openebs

davidkarlsen commented 3 years ago

This should be of interest:

docker run --rm -ti --entrypoint /bin/sh k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0
docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: exec: "/bin/sh": stat /bin/sh: no such file or directory: unknown.

and from the pod:

    image: k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0
    imagePullPolicy: IfNotPresent
    lifecycle:
      preStop:
        exec:
          command:
          - /bin/sh
          - -c
          - rm -rf /registration/lvm-localpv /registration/lvm-localpv-reg.sock
    name: csi-node-driver-registrar

i.e. the lifecycle-hook does not run!

kmova commented 3 years ago

@davidkarlsen -- the above information is really helpful! container permissions and location where the configuration is saved do sound like the possible causes for this issue.

Just spoke to @pawanpraka1 - and he wants to test out a few things. Will get back to this thread in a bit.

pawanpraka1 commented 3 years ago

i.e. the lifecycle-hook does not run!

That is a good find, but should not cause the issue. There will be stale socket file present on the system because of that.

@davidkarlsen can you execute pvscan --cache (and lvscan) command before doing node reboot. I am wondering whether node cache is not updated before reboot, if not, we might lose the volumes.

davidkarlsen commented 3 years ago

@davidkarlsen can you execute pvscan --cache (and lvscan) command before doing node reboot. I am wondering whether node cache is not updated before reboot, if not, we might lose the volumes.

It seems to not persist:

oco02 ~]# pvscan --cache
[root@alt-ksx-g-c01oco02 ~]# lvscan 
  ACTIVE            '/dev/datavg/openebs' [5.00 GiB] inherit
  ACTIVE            '/dev/datavg/pvc-6e5a7f2f-fc50-443e-859d-d6f76de94064' [1.00 GiB] inherit
  ACTIVE            '/dev/rootvg/swap_lv' [2.00 GiB] inherit
  ACTIVE            '/dev/rootvg/tmp_lv' [<8.79 GiB] inherit
  ACTIVE            '/dev/rootvg/var_lv' [150.00 GiB] inherit
  ACTIVE            '/dev/rootvg/home_lv' [1000.00 MiB] inherit
  ACTIVE            '/dev/rootvg/root_lv' [<6.84 GiB] inherit
[root@alt-ksx-g-c01oco02 ~]# cd /etc/
[root@alt-ksx-g-c01oco02 etc]# cd lvm/
[root@alt-ksx-g-c01oco02 lvm]# find . -type f|xargs fgrep -il  vc-6e5a7f2f-fc50-443e-859d-d6f76
[root@alt-ksx-g-c01oco02 lvm]#

davidkarlsen commented 3 years ago

Ok, so I've tried creating a bunch of volumes and rebooting machines. Two changes:

I noticed the previous openebs install was not 100%, because in some upgrade the storageCapacity flag went from false to true, so the helm install was not complete, as this flag is immutable. I deleted the csidriver and let the chart install it again which then succeeded.
I previously used XFS-based filesystems, these failed formatting by openebs due to https://github.com/openebs/lvm-localpv/issues/135 (which is really a kubernetes issue) - so these volumes were formatted by hand, now I use ext4 across the line.

I haven't seen the problem since, so I think we close this and I'll reopen if I can provoke it consistently.

npu21 commented 2 years ago

We occur this issue too!

After a reboot, we found a lv disappear while other lvs are normal. R`6 M7ZPTIVO}V 86A)`R2

The lost lv coundn't be found in /dev/mapper or /dev/lvmvg. There isn't any clue in the openebs pods logs or the lvm services

And found one wired thing is that there's no record of pvc-070f0023-d09e-4041-9cb9-dbd1a579e717 in /etc/lvm/archive. But the lv is surely created successfully and the pod worked well before restart. pvscan --cache return no error

root@master1:/etc/lvm/archive# lvm version
  LVM version:     2.02.176(2) (2017-11-03)
  Library version: 1.02.145 (2017-11-03)
  Driver version:  4.41.0
  Configuration:   ./configure --build=x86_64-linux-gnu --prefix=/usr --includedir=${prefix}/include --mandir=${prefix}/share/man --infodir=${prefix}/share/info --sysconfdir=/etc --localstatedir=/var --disable-silent-rules --libdir=${prefix}/lib/x86_64-linux-gnu --libexecdir=${prefix}/lib/x86_64-linux-gnu --runstatedir=/run --disable-maintainer-mode --disable-dependency-tracking --exec-prefix= --bindir=/bin --libdir=/lib/x86_64-linux-gnu --sbindir=/sbin --with-usrlibdir=/usr/lib/x86_64-linux-gnu --with-optimisation=-O2 --with-cache=internal --with-clvmd=corosync --with-cluster=internal --with-device-uid=0 --with-device-gid=6 --with-device-mode=0660 --with-default-pid-dir=/run --with-default-run-dir=/run/lvm --with-default-locking-dir=/run/lock/lvm --with-thin=internal --with-thin-check=/usr/sbin/thin_check --with-thin-dump=/usr/sbin/thin_dump --with-thin-repair=/usr/sbin/thin_repair --enable-applib --enable-blkid_wiping --enable-cmdlib --enable-cmirrord --enable-dmeventd --enable-dbus-service --enable-lvmetad --enable-lvmlockd-dlm --enable-lvmlockd-sanlock --enable-lvmpolld --enable-notify-dbus --enable-pkgconfig --enable-readline --enable-udev_rules --enable-udev_sync

Could anyone take a look at this serious issue? Thanks in advance!

npu21 commented 2 years ago

@davidkarlsen Could you reopen this issue?

npu21 commented 2 years ago

@kmova @pawanpraka1 Could you please take a look?

davidkarlsen commented 2 years ago

@davidkarlsen Could you reopen this issue?

reopened

davidkarlsen commented 2 years ago

@npu21 what OS do you run on the nodes?

npu21 commented 2 years ago

@davidkarlsen Thanks! Linux version 5.4.61-050461-generic (kernel@kathleen) (gcc version 9.3.0 (Ubuntu 9.3.0-10ubuntu2)) #202008260931 SMP Wed Aug 26 09:34:29 UTC 2020

npu21 commented 2 years ago

one more clue is the lvm logs

gunzy83 commented 2 years ago

I am currently evaluating some local storage options and reading through issues here and elsewhere but thought I might chime in. This is sounding like a case of openebs-lvm-node not having an exclusive lock on the metadata of the volume group (ie lvm2-lvmmetad.service is interfering). I do not think it is a good idea having multiple services operating on the volume group at once. Could you try to run some tests with lvm2-lvmmetad.service stopped/disabled? It is not even technically required to be running for the pvcreate and vgcreate steps in the install instructions.

npu21 commented 2 years ago

@gunzy83 Agree! There seems need to add an exclusive lock on createing and delete logic volumes @wrtabhi611 Could you please take a look at this ? When going through the source code, I found there are places operating pvs,vgs,lvs supposing atomic operation, which are not in fact. Thanks in advance!

gunzy83 commented 2 years ago

@gunzy83 Agree! There seems need to add an exclusive lock on createing and delete logic volumes

I wonder if lvm2-lvmmetad.service operates with the same semantics (assuming it has exclusive use of the pvs/vgs/lvs)? If so adding logic into openebs would be the way to go. If not, providing some general guidance in the documentation that openebs-lvm-node needs exclusive use would be prudent.

davidkarlsen commented 2 years ago

Why would this be needed? Doesn't the service run normally, and users can run LVM commands without LVM getting corrupt?

gunzy83 commented 2 years ago

Only a theory but users on the host would be running those commands from the same namespace I assume. Inside a container you would need to break namespace isolation for commands/services on the host to be able to coordinate with the commands being run from the container. Would be happy to be proven wrong.

For reference, Topolvm look to have solved the issue they had, they were running their lvmd on the host to avoid it until they implemented a solution using nsenter and can now run it in a daemonset. carina.io also use nsenter to escape isolation but I can't find any issues on Github to follow that show how they came to that.

npu21 commented 2 years ago

@davidkarlsen @gunzy83 In fact, we have reproduced this issue more than once. Procedure

These 9 pvcs have to occupy the vg full
1. Execute mkdir -p /root/lvmtest/pvc to create 9 pvc yaml files, execute kubectl apply -f /root/lvmtest/pvc/ to create 9 pvcs, 9 pvcs will be the vg where lvm is located full
2. Execute mkdir -p /root/lvmtest/pod to create 9 pod yaml files, execute kubectl apply -f /root/lvmtest/pod/ to create 9 pods and mount the pvc created in the previous step respectively
3. Execute kubectl get pvc to see that all 9 pvcs are Bound
4. Execute lvs to see if there are 9 corresponding lvs
5. Execute kubectl delete -f /root/lvmtest/pod/, kubectl delete -f /root/lvmtest/pvc/, delete pod and pvc, and then execute kubectl get pvc to see if there is any pvc
6. Execute kubectl apply -f /root/lvmtest/pvc/ to create 9 pvcs
7. Execute kubectl apply -f /root/lvmtest/pod/ to create 9 pods and mount the pvc created in the previous step respectively
8.1. Execute kubectl delete lvmvolume -n kube-system pvc-xxx, delete the lv created in the first round in order from small to large
8.2. Execute lvremove, delete the lv created in the first round in order from small to large
9. Execute kubectl get pvc to check whether the 9 pvcs are Bound
10. Execute lvs to see if there are 9 corresponding lvs, execute ls /dev/mapper, ls /dev/lvmvg to see if there are 9 corresponding lvm directories

After runing above steps circularly ,there are chances to reproduct the issue. The lvs will display <9 LVs , and the lost LV will disappear forever after reboot.

npu21 commented 2 years ago

@davidkarlsen Did you notice this locking_type in /etc/lvm/lvm.conf ? My test env didn't config this previously. I've configured this and retest now

tmsdce commented 2 years ago

I think @gunzy83 is right regarding lvmetad interfering with openebs in the handling of LVs (a locking issue seems indeed a good place to start investigating). This seems confirmed in my testings too.

I tried to use lvm-localpv on Flatcar Linux and nothing worked as expected (I know it's not officially supported but that's what I'm using ^^). The controller was happy with volumes being created but no LVs showed up on the host (not even after a pvscan --cache). After rebooting the node, even the controller complained about LVs not being found as if the host couldn't persist the metadata in some way.

On Ubuntu 20.04, everything worked fine, so I compared the related LVM services started on the two different OS's. It turned out that lvm2-lvmetad was started on Flatcar (via a systemd socket) and not on Ubuntu.

As @gunzy83 suggested above, I simply disabled and masked the lvm2-metad service and socket on Flatcar and from then on everything seems to work as expected. Openebs correctly creates the volumes, lvs command on Flatcar host correctly displays the related LVs (no need for pvscan --cache, LVs are kept after reboots, symlinks in /dev/disk/ are handled correctly as well. Since lvmetad service is no longer active, all LVM-related commands and actions need to use device scanning. This may have an impact on performance if using many devices and may be a concern for some users, so having the plugin playing nice with lvmetad could be really great.

dsharma-dc commented 6 months ago

It is an old thread but I'd like to point out that the lvmetad daemon has been deprecated now. Now LVM will always read metadata from disk. This would mean the LVs should get autoactivated,a behaviour which was earlier linked to use_lvmetad. For older distros, disabling lvm2-metad service should serve as a help as suggested above.

dsharma-dc commented 6 months ago

I am closing this issue now as it is reasonably understood to be the issue in metadata persistence during reboot when lvmetad daemon is enabled. Please feel free to reopen if still have any concerns around this.

openebs / lvm-localpv

Volumes go missing. #157