nfs server start failed

nAnderYang commented 1 year ago

i change nfsd.sh in nfs-server-alpine:0.9.0 image. add line ulimit -HSn 1048576 to nfsd.sh fix the problem.

    echo "Starting Mountd in the background..."These
    ulimit -HSn 1048576 #add line
    /usr/sbin/rpc.mountd --debug all --no-udp --no-nfs-version 2 --no-nfs-version 3

create rwx pvc the nfs pod loop error:

Displaying /etc/exports contents:
/nfsshare *(rw,fsid=0,async,no_subtree_check,no_auth_nlm,insecure,no_root_squash)

Starting rpcbind...
rpcbind: another rpcbind is already running. Aborting
Displaying rpcbind status...
   program version netid     address                service    owner
    100000    4    tcp6      ::.0.111               -          superuser
    100000    3    tcp6      ::.0.111               -          superuser
    100000    4    udp6      ::.0.111               -          superuser
    100000    3    udp6      ::.0.111               -          superuser
    100000    4    tcp       0.0.0.0.0.111          -          superuser
    100000    3    tcp       0.0.0.0.0.111          -          superuser
    100000    2    tcp       0.0.0.0.0.111          -          superuser
    100000    4    udp       0.0.0.0.0.111          -          superuser
    100000    3    udp       0.0.0.0.0.111          -          superuser
    100000    2    udp       0.0.0.0.0.111          -          superuser
    100000    4    local     /var/run/rpcbind.sock  -          superuser
    100000    3    local     /var/run/rpcbind.sock  -          superuser
    100003    4    tcp       0.0.0.0.8.1            -          superuser
    100003    4    tcp6      ::.8.1                 -          superuser
Starting NFS in the background...
rpc.nfsd: knfsd is currently up
Exporting File System...
exporting *:/nfsshare
/nfsshare       <world>
Starting Mountd in the background...These
rpc.mountd: setrlimit (RLIMIT_NOFILE) failed: Operation not permitted
Startup of NFS failed, sleeping for 2s, then retrying...

kubectl get pods -n openebs --show-labels

NAME                                                              READY   STATUS    RESTARTS       AGE     LABELS
cspc-operator-85d95555c-pp8t6                                     1/1     Running   2 (34m ago)    8d      name=cspc-operator,openebs.io/component-name=cspc-operator,openebs.io/version=3.3.0,pod-template-hash=85d95555c
cstor-storage-8l5t-75fffc7f94-pksht                               3/3     Running   6 (34m ago)    8d      app=cstor-pool,openebs.io/cstor-pool-cluster=cstor-storage,openebs.io/cstor-pool-instance=cstor-storage-8l5t,openebs.io/version=3.3.0,pod-template-hash=75fffc7f94
cstor-storage-dz8g-f7d4f5fcc-k95xn                                3/3     Running   6 (34m ago)    8d      app=cstor-pool,openebs.io/cstor-pool-cluster=cstor-storage,openebs.io/cstor-pool-instance=cstor-storage-dz8g,openebs.io/version=3.3.0,pod-template-hash=f7d4f5fcc
cstor-storage-m6dw-6499b4c96f-tgqxw                               3/3     Running   6 (34m ago)    8d      app=cstor-pool,openebs.io/cstor-pool-cluster=cstor-storage,openebs.io/cstor-pool-instance=cstor-storage-m6dw,openebs.io/version=3.3.0,pod-template-hash=6499b4c96f
cvc-operator-9f4bfc89-dkmnn                                       1/1     Running   2 (34m ago)    8d      name=cvc-operator,openebs.io/component-name=cvc-operator,openebs.io/version=3.3.0,pod-template-hash=9f4bfc89
nfs-pvc-416f95f7-559d-4ae4-a7df-3a00f4c22fbc-7f7cc94f4f-q5w7j     1/1     Running   1 (10m ago)    31m     openebs.io/nfs-server=nfs-pvc-416f95f7-559d-4ae4-a7df-3a00f4c22fbc,pod-template-hash=7f7cc94f4f
openebs-cstor-admission-server-75df668b7-n6g4c                    1/1     Running   2 (34m ago)    8d      app=cstor-admission-webhook,openebs.io/component-name=cstor-admission-webhook,openebs.io/version=3.3.0,pod-template-hash=75df668b7
openebs-cstor-csi-controller-0                                    6/6     Running   13 (34m ago)   8d      app=openebs-cstor-csi-controller,controller-revision-hash=openebs-cstor-csi-controller-7c6b54bbbc,name=openebs-cstor-csi-controller,openebs.io/component-name=openebs-cstor-csi-controller,openebs.io/version=3.3.0,role=openebs-cstor-csi,statefulset.kubernetes.io/pod-name=openebs-cstor-csi-controller-0
openebs-cstor-csi-node-9v78c                                      2/2     Running   4 (34m ago)    8d      app=openebs-cstor-csi-node,controller-revision-hash=56945c4cd9,name=openebs-cstor-csi-node,openebs.io/component-name=openebs-cstor-csi-node,openebs.io/version=3.3.0,pod-template-generation=1,role=openebs-cstor-csi
openebs-cstor-csi-node-fkmqg                                      2/2     Running   4 (34m ago)    8d      app=openebs-cstor-csi-node,controller-revision-hash=56945c4cd9,name=openebs-cstor-csi-node,openebs.io/component-name=openebs-cstor-csi-node,openebs.io/version=3.3.0,pod-template-generation=1,role=openebs-cstor-csi
openebs-cstor-csi-node-tz8nq                                      2/2     Running   4 (34m ago)    8d      app=openebs-cstor-csi-node,controller-revision-hash=56945c4cd9,name=openebs-cstor-csi-node,openebs.io/component-name=openebs-cstor-csi-node,openebs.io/version=3.3.0,pod-template-generation=1,role=openebs-cstor-csi
openebs-ndm-5bjcl                                                 1/1     Running   3 (34m ago)    8d      controller-revision-hash=c7c95b565,name=openebs-ndm,openebs.io/component-name=ndm,openebs.io/version=3.3.0,pod-template-generation=1
openebs-ndm-64kvp                                                 1/1     Running   3 (33m ago)    8d      controller-revision-hash=c7c95b565,name=openebs-ndm,openebs.io/component-name=ndm,openebs.io/version=3.3.0,pod-template-generation=1
openebs-ndm-bq6s9                                                 1/1     Running   3 (34m ago)    8d      controller-revision-hash=c7c95b565,name=openebs-ndm,openebs.io/component-name=ndm,openebs.io/version=3.3.0,pod-template-generation=1
openebs-ndm-cluster-exporter-6f4c89665f-5lzqg                     1/1     Running   2 (34m ago)    8d      name=openebs-ndm-cluster-exporter,openebs.io/component-name=ndm-cluster-exporter,openebs.io/version=3.3.0,pod-template-hash=6f4c89665f
openebs-ndm-node-exporter-4fbqd                                   1/1     Running   2 (34m ago)    8d      controller-revision-hash=678d8684cd,name=openebs-ndm-node-exporter,openebs.io/component-name=ndm-node-exporter,openebs.io/version=3.3.0,pod-template-generation=1
openebs-ndm-node-exporter-c77qq                                   1/1     Running   2 (34m ago)    8d      controller-revision-hash=678d8684cd,name=openebs-ndm-node-exporter,openebs.io/component-name=ndm-node-exporter,openebs.io/version=3.3.0,pod-template-generation=1
openebs-ndm-node-exporter-g5tqr                                   1/1     Running   2 (34m ago)    8d      controller-revision-hash=678d8684cd,name=openebs-ndm-node-exporter,openebs.io/component-name=ndm-node-exporter,openebs.io/version=3.3.0,pod-template-generation=1
openebs-ndm-operator-696c55bfff-kj4wn                             1/1     Running   3 (34m ago)    8d      name=openebs-ndm-operator,openebs.io/component-name=ndm-operator,openebs.io/version=3.3.0,pod-template-hash=696c55bfff
openebs-nfs-provisioner-6f5b7dcbc8-kchlk                          1/1     Running   2 (34m ago)    8d      name=openebs-nfs-provisioner,openebs.io/component-name=openebs-nfs-provisioner,openebs.io/version=0.9.0,pod-template-hash=6f5b7dcbc8
pvc-7b50b611-c7e3-4406-b4a7-d9197434c504-target-5fc8d84f9cpxxm7   3/3     Running   10 (33m ago)   7d16h   app=cstor-volume-manager,monitoring=volume_exporter_prometheus,openebs.io/persistent-volume-claim=test-r-w-o,openebs.io/persistent-volume=pvc-7b50b611-c7e3-4406-b4a7-d9197434c504,openebs.io/target=cstor-target,openebs.io/version=3.3.0,pod-template-hash=5fc8d84f9c
pvc-b5c5a903-6970-45c2-aa65-fdb923bc2d36-target-6d97d95dc7j7flf   3/3     Running   10 (33m ago)   7d16h   app=cstor-volume-manager,monitoring=volume_exporter_prometheus,openebs.io/persistent-volume-claim=nfs-pvc-416f95f7-559d-4ae4-a7df-3a00f4c22fbc,openebs.io/persistent-volume=pvc-b5c5a903-6970-45c2-aa65-fdb923bc2d36,openebs.io/target=cstor-target,openebs.io/version=3.3.0,pod-template-hash=6d97d95dc7

kubectl get pvc -n openebs

NAME                                           STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
nfs-pvc-416f95f7-559d-4ae4-a7df-3a00f4c22fbc   Bound    pvc-b5c5a903-6970-45c2-aa65-fdb923bc2d36   10Gi       RWO            cstor-csi      7d16h
nfs-pvc-a84b182f-d658-4ae4-9c67-0093867c5a6e   Bound    pvc-d63176c3-643f-439b-ad86-d09d02ceba35   10Gi       RWO            cstor-csi      8d

kubectl get pvc -n default

NAME         STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
test-r-w-o   Bound    pvc-7b50b611-c7e3-4406-b4a7-d9197434c504   10Gi       RWO            cstor-csi      7d16h
test-r-w-x   Bound    pvc-416f95f7-559d-4ae4-a7df-3a00f4c22fbc   10Gi       RWX            cstor-nfs      7d16h

Environment details:

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1", GitCommit:"86ec240af8cbd1b60bcc4c03c20da9b98005b92e", GitTreeState:"clean", BuildDate:"2021-12-16T11:41:01Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.1", GitCommit:"86ec240af8cbd1b60bcc4c03c20da9b98005b92e", GitTreeState:"clean", BuildDate:"2021-12-16T11:34:54Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}

hardware configuration: test vm 4C4T, 8G RAM 200G/OS 200G/DATA block

OS:

NAME="Rocky Linux"
VERSION="9.0 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.0"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Rocky Linux 9.0 (Blue Onyx)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:9::baseos"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9"
ROCKY_SUPPORT_PRODUCT_VERSION="9.0"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.0"

kernel: 5.14.0-70.22.1.el9_0.x86_64
containerd: containerd containerd.io 1.6.8 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6

dsharma-dc commented 12 months ago

This looks like a platform difference in the file descriptor limits. What was the default ulimit -n value before changing it fixed the mountd startup issue?

nAnderYang commented 12 months ago

That test environment has been destroy long time ago. I can't get the value. This issue only reporting the information.

dsharma-dc commented 12 months ago

Thank you for responding. I understand this has been open for quite a long time. I'd be closing this issue now as this looks specific to the OS platform default configuration. Please feel free to revert back and reopen if there is a need.

openebs-archive / dynamic-nfs-provisioner

nfs server start failed #149