Delay in pod deletion when having localpv provisioned

[lordofire]

Describe the bug: When a pod is created with the localpv provisioned, deleting the pod using kubectl along with the pv will take around 40+ seconds whereas creating a pod should return instantly.

Expected behaviour: Delete a pod should have very small blocking wait compared to pod creation.

Steps to reproduce the bug:

1. Setup the hostpath with xfs quota enabled. Install the openebs localpv provisioner.
2. Create a pod along with the pv with the following configuration test.yaml

   apiVersion: apps/v1
   kind: List
   items:
   - kind: PersistentVolumeClaim
   apiVersion: v1
   metadata:
   name: local-hostpath-pvc
   namespace: infra-offline-dev
   spec:
   storageClassName: openebs-hostpath
   accessModes:
     - ReadWriteOnce
   resources:
     requests:
       storage: 5G
   - apiVersion: v1
   kind: Pod
   metadata:
   name: hello-local-device-pod
   namespace: infra-offline-dev
   spec:
   volumes:
     - name: local-storage
       persistentVolumeClaim:
         claimName: local-hostpath-pvc
   containers:
     - name: hello-container
       image: busybox
       command:
         - sh
         - -c
         - 'while true; do echo "`date` [`hostname`] Hello from OpenEBS Local PV." >> /mnt/store/greet.txt; sleep $(($RANDOM % 5 + 300)); done'
       volumeMounts:
         - mountPath: /mnt/store
           name: local-storage

   cmd: kubectl apply -f test.yaml

3. After waiting the pod being created, delete the pod as well as the pvc together using kubectl: cmd: kubectl delete -f test.yaml

My test output when performing the delete:

Linux$ time kubectl delete -f test.yaml
persistentvolumeclaim "local-hostpath-pvc" deleted
pod "hello-local-device-pod" deleted

real    0m43.641s
user    0m0.098s
sys 0m0.034s

Anything else we need to know?: N/A

Environment details:

• OpenEBS version (use kubectl get po -n openebs --show-labels):

  NAME                                            READY   STATUS    RESTARTS   AGE   LABELS
  openebs-localpv-provisioner-5d88cb474b-j59qk    1/1     Running   16         32d   name=openebs-localpv-provisioner,openebs.io/component-name=openebs-localpv-provisioner,openebs.io/version=3.3.0,pod-template-hash=5d88cb474b
  openebs-ndm-cluster-exporter-84bb5fc764-jxtkh   1/1     Running   0          32d   name=openebs-ndm-cluster-exporter,openebs.io/component-name=ndm-cluster-exporter,openebs.io/version=3.3.0,pod-template-hash=84bb5fc764
  openebs-ndm-g2rsl                               1/1     Running   0          31d   controller-revision-hash=56bdc87c48,name=openebs-ndm,openebs.io/component-name=ndm,openebs.io/version=3.3.0,pod-template-generation=3
  openebs-ndm-node-exporter-6wt6d                 1/1     Running   0          31d   controller-revision-hash=5fc9bcf946,name=openebs-ndm-node-exporter,openebs.io/component-name=ndm-node-exporter,openebs.io/version=3.3.0,pod-template-generation=3
  openebs-ndm-operator-7657446466-cz4w9           1/1     Running   0          32d   name=openebs-ndm-operator,openebs.io/component-name=ndm-operator,openebs.io/version=3.3.0,pod-template-hash=7657446466

• Kubernetes version (use kubectl version):

  Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.12", GitCommit:"696a9fdd2a58340e61e0d815c5769d266fca0802", GitTreeState:"clean", BuildDate:"2022-04-13T19:07:00Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.12", GitCommit:"696a9fdd2a58340e61e0d815c5769d266fca0802", GitTreeState:"clean", BuildDate:"2022-04-13T19:01:10Z", GoVersion:"go1.16.15", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration: Bare-metal. XFS config is the following:

  UUID="1e99f408-4103-4432-b5c0-f943400c8e6a" /data         xfs     defaults,uquota,pquota 0   0
  /data/k8s/pv /var/kubernetes/pv auto bind 0 0

• OS (e.g: cat /etc/os-release):

  NAME="Ubuntu"
  VERSION="20.04.5 LTS (Focal Fossa)"
  ID=ubuntu
  ID_LIKE=debian
  PRETTY_NAME="Ubuntu 20.04.5 LTS"
  VERSION_ID="20.04"
  HOME_URL="https://www.ubuntu.com/"
  SUPPORT_URL="https://help.ubuntu.com/"
  BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
  PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
  VERSION_CODENAME=focal
  UBUNTU_CODENAME=focal

• kernel (e.g: uname -a):

  Linux test-host 5.15.0-46-generic #49~20.04.1-Ubuntu SMP Thu Aug 4 19:15:44 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

• others: N/A

[lordofire]

Any update on this issue?

[lordofire]

Any update on this issue?

[niladrih]

This might help in achieving quicker Pod-schedule times for the hostpath delete job. There aren't a lot of tweak-able setting that make the controller more. However, that is an enhancement that could benefit users.

I'll see the hostNetwork change going in in v4.1.0. Maybe you could see if that helps with your times.

[tiagolobocastro]

@lordofire did you capture some logs from openebs-localpv-provisioner-5d88cb474b-j59qk at the time? It seems to have restarted 16 times. @niladrih any updates here, do you understand the root cause?