Closed sjerman closed 1 month ago
Hey @sjerman, thanks for reporting this. Here's what I did and was able to provision PVC using local-lvm storage class. Do tell me what you did differently here:
sudo snap install microk8s --classic --channel=1.27
$ k get pvc -A
NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
default csi-lvmpv Bound pvc-08e2c003-ae88-4f40-867a-b4d9f5432487 4Gi RWO openebs-lvmpv 5s
What steps and/or configuration did you follow that were different?
Hi,
spent a little time on it. It seems at though the issue happens with nfs-provisioning via the lvm provisioner.
Steve
So :)
snap install microk8s --classic --channel=1.27/stable
microk8s helm repo add openebs https://openebs.github.io/charts
microk8s helm install --create-namespace -n openebs --wait -f openebs.yaml openebs openebs/openebs
microk8s kubectl install -f config.yaml
And you get symptoms like:
jerman@steve-nuc:~/Projects/microk8s-bug$ kubectl describe pod -n openebs nfs-pvc-7ae6fe71-1dcd-43bc-876e-9fe7f6cf966c-645597df89-dwftx
Name: nfs-pvc-7ae6fe71-1dcd-43bc-876e-9fe7f6cf966c-645597df89-dwftx
Namespace: openebs
...
Labels: nfs.openebs.io/nfs-pvc-name=nfs-rwx-pvc
nfs.openebs.io/nfs-pvc-namespace=default
nfs.openebs.io/nfs-pvc-uid=7ae6fe71-1dcd-43bc-876e-9fe7f6cf966c
openebs.io/nfs-server=nfs-pvc-7ae6fe71-1dcd-43bc-876e-9fe7f6cf966c
pod-template-hash=645597df89
...
Volumes:
exports-dir:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: nfs-pvc-7ae6fe71-1dcd-43bc-876e-9fe7f6cf966c
ReadOnly: false
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 3m12s default-scheduler 0/1 nodes are available: 1 node(s) did not have enough free storage. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling..
Doesn't happen on 1.26.
Relevant files....
Oh.. one thing.. you will need to edit the config.yaml file to set the volume group to use. The default is vgubuntu
.. and it assigns 2x 10G volumes...
Hi @sjerman, Thanks for the steps, I was able to replicate your error. The problem probably occurs during the external provisioning of PVs. The logs are full with
waiting for a volume to be created, either by external provisioner "openebs.io/nfsrwx" or manually created by system administrator
I tried to isolate the entire thing to see if the problem was with the NFS provisioned. We deploy the NFS provisioner while deploying the helm chart via the configs so that's already done. Then I created the nfs rwx storage class. But after that, when I create a sample PVC, it remains stuck in the pending
state with logs:
│ Type Reason Age From Message │
│ ---- ------ ---- ---- ------- │
│ Normal Provisioning 4m45s (x7 over 17m) openebs.io/nfsrwx_openebs-nfs-provisioner-74f4f7cffd-2z72t_a8d3d535-64be-40af-a297-904bbef73094 External provisioner is provisioning volume for claim "default/nfs-pvc" │
│ Warning ProvisioningFailed 3m45s (x7 over 16m) openebs.io/nfsrwx_openebs-nfs-provisioner-74f4f7cffd-2z72t_a8d3d535-64be-40af-a297-904bbef73094 failed to provision volume with StorageClass "nfs-rwx-storage": failed to deploy NFS Server: timed out waiting for PVC{openebs/nfs-pvc-05a685ce-ec56- │
│ 45f7-bcd3-83be452ffdc6} to bound │
│ Normal ExternalProvisioning 2m24s (x63 over 17m) persistentvolume-controller waiting for a volume to be created, either by external provisioner "openebs.io/nfsrwx" or manually created by system administrator │
│ ```
Searchin a bit more, I found this issue in openEBS dynamic nfs provisioner which is similar to our issue. I've commented there to get the discussion started around this issue for microk8s.
Hi @sachinkumarsingh092 By any chance don't you know - is there a solution for that? I'm sort of stuck with the problem when new PVCscan't be created after the upgrade. Getting: waiting for a volume to be created, either by external provisioner "openebs.io/local" or manually created by system administrator
The problem is that I can't even provision volumes using standard hostpath class.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Summary
When you install the openebs local-lvm helm chart on MicrosK8S 1.27.2 revision 5372. PVC provisioned using a local-lvm storage class get stuck in 'pending'.
if you change the version of MicroK8S to 1.26.2 everything starts working again..
What Should Happen Instead?
PVC should be provisioned.
Reproduction Steps
We have duplicated the issue on multiple systems.
Introspection Report
attached
Can you suggest a fix?
Are you interested in contributing with a fix?
inspection-report-20230607_094320.tar.gz