kvaps
commented
2 years ago To reproduce:
#!/bin/sh
kubectl delete sc piraeus-ssd
for INSTANCE in $(seq 1 100); do
kubectl create -f- <<EOT
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: my-pvc${INSTANCE}
labels:
app: "test"
spec:
accessModes:
- ReadWriteOnce
volumeMode: Block
storageClassName: piraeus-ssd
resources:
requests:
storage: 10Gi
---
apiVersion: v1
kind: Pod
metadata:
name: my-pod${INSTANCE}
labels:
app: "test"
spec:
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- test
topologyKey: "kubernetes.io/hostname"
containers:
- name: my-container
image: alpine:3.14
imagePullPolicy: IfNotPresent
command:
- sleep
- infinity
volumeDevices:
- devicePath: /dev/xvda
name: my-volume
volumes:
- name: my-volume
persistentVolumeClaim:
claimName: my-pvc${INSTANCE}
terminationGracePeriodSeconds: 0
EOT
done
kubectl create -f- <<EOT
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: piraeus-ssd
parameters:
autoPlace: "2"
storagePool: lvm
provisioner: linstor.csi.linbit.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
EOT
JoelColledge
Hi I just faced the bug and I think I know how to reproduce it
I have LINSTOR on three nodes and 2xNVME on each of them.
OS: Ubuntu 20.04.3 LTS kernel: 5.13.0-27-generic DRBD version: 9.1.4 (api:2/proto:110-121) LINSTOR version: 1.17.0
LINSTOR installed using priaeus-operator.
I created three LVM pools:
And 51 pod with 10GB volumes on each of them:
48 volumes provisioned without problems, but three of them are stuck on Inconsistent state:
I fixed these devices by running:
linstor-controller log is full of these messages even for other successfully created resources:
linstor err show 61F188C3-00000-000008
ERROR REPORT 61F188C3-00000-000008 ============================================================ Application: LINBIT�� LINSTOR Module: Controller Version: 1.17.0 Build ID: 7e646d83dbbadf1ec066e1bc8b29ae018aff1f66 Build time: 2021-12-09T07:27:52+00:00 Error time: 2022-01-26 19:18:11 Node: piraeus-op-cs-controller-554c4d7fc8-bqskt Peer: RestClient(10.111.2.166; 'linstor-csi/') ============================================================ Reported error: =============== Category: RuntimeException Class name: ApiRcException Class canonical name: com.linbit.linstor.core.apicallhandler.response.ApiRcException Generated at: Method 'autoPlaceInTransaction', Source file 'CtrlRscAutoPlaceApiCallHandler.java', Line #235 Error message: The resource 'pvc-b442c3dc-d7b4-4ba3-9755-d47351bf4d42' was already deployed on 3 nodes: 'hf-kubevirt-01', 'hf-kubevirt-02', 'hf-kubevirt-03'. The resource would have to be deleted from nodes to reach the placement count. Error context: The resource 'pvc-b442c3dc-d7b4-4ba3-9755-d47351bf4d42' was already deployed on 3 nodes: 'hf-kubevirt-01', 'hf-kubevirt-02', 'hf-kubevirt-03'. The resource would have to be deleted from nodes to reach the placement count. Asynchronous stage backtrace: Error has been observed at the following site(s): |_ checkpoint ? Auto-place resource Stack trace: Call backtrace: Method Native Class:Line number autoPlaceInTransaction N com.linbit.linstor.core.apicallhandler.controller.CtrlRscAutoPlaceApiCallHandler:235 Suppressed exception 1 of 1: =============== Category: RuntimeException Class name: OnAssemblyException Class canonical name: reactor.core.publisher.FluxOnAssembly.OnAssemblyException Generated at: Method 'autoPlaceInTransaction', Source file 'CtrlRscAutoPlaceApiCallHandler.java', Line #235 Error message: Error has been observed at the following site(s): |_ checkpoint ��� Auto-place resource Stack trace: Error context: The resource 'pvc-b442c3dc-d7b4-4ba3-9755-d47351bf4d42' was already deployed on 3 nodes: 'hf-kubevirt-01', 'hf-kubevirt-02', 'hf-kubevirt-03'. The resource would have to be deleted from nodes to reach the placement count. Call backtrace: Method Native Class:Line number autoPlaceInTransaction N com.linbit.linstor.core.apicallhandler.controller.CtrlRscAutoPlaceApiCallHandler:235 lambda$autoPlace$0 N com.linbit.linstor.core.apicallhandler.controller.CtrlRscAutoPlaceApiCallHandler:146 doInScope N com.linbit.linstor.core.apicallhandler.ScopeRunner:147 lambda$fluxInScope$0 N com.linbit.linstor.core.apicallhandler.ScopeRunner:75 call N reactor.core.publisher.MonoCallable:91 trySubscribeScalarMap N reactor.core.publisher.FluxFlatMap:126 subscribeOrReturn N reactor.core.publisher.MonoFlatMapMany:49 subscribe N reactor.core.publisher.Flux:8343 onNext N reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:188 request N reactor.core.publisher.Operators$ScalarSubscription:2344 onSubscribe N reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:134 subscribe N reactor.core.publisher.MonoCurrentContext:35 subscribe N reactor.core.publisher.Flux:8357 onNext N reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:188 onNext N reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber:121 complete N reactor.core.publisher.Operators$MonoSubscriber:1782 onComplete N reactor.core.publisher.MonoCollect$CollectSubscriber:152 onComplete N reactor.core.publisher.MonoFlatMapMany$FlatMapManyInner:252 checkTerminated N reactor.core.publisher.FluxFlatMap$FlatMapMain:838 drainLoop N reactor.core.publisher.FluxFlatMap$FlatMapMain:600 drain N reactor.core.publisher.FluxFlatMap$FlatMapMain:580 onComplete N reactor.core.publisher.FluxFlatMap$FlatMapMain:457 checkTerminated N reactor.core.publisher.FluxFlatMap$FlatMapMain:838 drainLoop N reactor.core.publisher.FluxFlatMap$FlatMapMain:600 innerComplete N reactor.core.publisher.FluxFlatMap$FlatMapMain:909 onComplete N reactor.core.publisher.FluxFlatMap$FlatMapInner:1013 onComplete N reactor.core.publisher.FluxMap$MapSubscriber:136 onComplete N reactor.core.publisher.Operators$MultiSubscriptionSubscriber:2016 onComplete N reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber:78 complete N reactor.core.publisher.FluxCreate$BaseSink:438 drain N reactor.core.publisher.FluxCreate$BufferAsyncSink:784 complete N reactor.core.publisher.FluxCreate$BufferAsyncSink:732 drainLoop N reactor.core.publisher.FluxCreate$SerializedSink:239 drain N reactor.core.publisher.FluxCreate$SerializedSink:205 complete N reactor.core.publisher.FluxCreate$SerializedSink:196 apiCallComplete N com.linbit.linstor.netcom.TcpConnectorPeer:455 handleComplete N com.linbit.linstor.proto.CommonMessageProcessor:363 handleDataMessage N com.linbit.linstor.proto.CommonMessageProcessor:287 doProcessInOrderMessage N com.linbit.linstor.proto.CommonMessageProcessor:235 lambda$doProcessMessage$3 N com.linbit.linstor.proto.CommonMessageProcessor:220 subscribe N reactor.core.publisher.FluxDefer:46 subscribe N reactor.core.publisher.Flux:8357 onNext N reactor.core.publisher.FluxFlatMap$FlatMapMain:418 drainAsync N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:414 drain N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:679 onNext N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:243 drainFused N reactor.core.publisher.UnicastProcessor:286 drain N reactor.core.publisher.UnicastProcessor:329 onNext N reactor.core.publisher.UnicastProcessor:408 next N reactor.core.publisher.FluxCreate$IgnoreSink:618 next N reactor.core.publisher.FluxCreate$SerializedSink:153 processInOrder N com.linbit.linstor.netcom.TcpConnectorPeer:373 doProcessMessage N com.linbit.linstor.proto.CommonMessageProcessor:218 lambda$processMessage$2 N com.linbit.linstor.proto.CommonMessageProcessor:164 onNext N reactor.core.publisher.FluxPeek$PeekSubscriber:177 runAsync N reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:439 run N reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:526 call N reactor.core.scheduler.WorkerTask:84 call N reactor.core.scheduler.WorkerTask:37 run N java.util.concurrent.FutureTask:264 run N java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask:304 runWorker N java.util.concurrent.ThreadPoolExecutor:1128 run N java.util.concurrent.ThreadPoolExecutor$Worker:628 run N java.lang.Thread:829 END OF ERROR REPORT.csi-controller log:
csi-provisioner
I0126 19:16:56.005108 1 controller.go:1279] provision "default/my-pvc10" class "piraeus-ssd": started I0126 19:16:56.005225 1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"my-pvc10", UID:"b442c3dc-d7b4-4ba3-9755-d47351bf4d42", APIVersion:"v1", ResourceVersion:"7196056", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/my-pvc10" W0126 19:17:56.005654 1 controller.go:933] Retrying syncing claim "b442c3dc-d7b4-4ba3-9755-d47351bf4d42", failure 0 E0126 19:17:56.005678 1 controller.go:956] error syncing claim "b442c3dc-d7b4-4ba3-9755-d47351bf4d42": failed to provision volume with StorageClass "piraeus-ssd": rpc error: code = DeadlineExceeded desc = context deadline exceeded I0126 19:17:56.005705 1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"my-pvc10", UID:"b442c3dc-d7b4-4ba3-9755-d47351bf4d42", APIVersion:"v1", ResourceVersion:"7196056", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "piraeus-ssd": rpc error: code = DeadlineExceeded desc = context deadline exceeded I0126 19:17:57.006240 1 controller.go:1279] provision "default/my-pvc10" class "piraeus-ssd": started I0126 19:17:57.006347 1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"my-pvc10", UID:"b442c3dc-d7b4-4ba3-9755-d47351bf4d42", APIVersion:"v1", ResourceVersion:"7196056", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/my-pvc10" I0126 19:18:25.116045 1 controller.go:1384] provision "default/my-pvc10" class "piraeus-ssd": volume "pvc-b442c3dc-d7b4-4ba3-9755-d47351bf4d42" provisioned I0126 19:18:25.116068 1 controller.go:1397] provision "default/my-pvc10" class "piraeus-ssd": succeeded I0126 19:18:25.118726 1 event.go:282] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"my-pvc10", UID:"b442c3dc-d7b4-4ba3-9755-d47351bf4d42", APIVersion:"v1", ResourceVersion:"7196056", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-b442c3dc-d7b4-4ba3-9755-d47351bf4d42linstor-csi-plugin
linstor-satellites log:
hf-kubevirt-01
hf-kubevirt-02
hf-kubevirt-03
dmesg from nodes:
hf-kubevirt-01
hf-kubevirt-02
hf-kubevirt-03