search

hpe-storage / csi-driver

A Container Storage Interface (CSI) driver from HPE
https://scod.hpedev.io
Apache License 2.0
55 stars 53 forks source link

NFS - Cannot acquire credentials for principal NFS #401

Closed nin0-0 closed 2 months ago

nin0-0 commented 2 months ago

hpe csi driver 2.4.2 ocp 4.12 installed through operator, working fine (seemingly) in 4 other clusters on 2 other alletra 9000, this one having issues

[root@bl1506 ticams003]# oc get pod -n hpe-nfs | grep a899b66f
hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw   1/1     Running                0          20h
[root@bl1506 ticams003]# oc logs -n hpe-nfs hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw
* Starting rpcbind
starting NFS server...
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] init_logging :LOG :NULL :LOG: Setting log level for all components to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_LOG from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_MEM_ALLOC from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_MEMLEAKS from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_FSAL from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_NFSPROTO from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_NFS_V4 from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_EXPORT from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_FILEHANDLE from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_DISPATCH from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_CACHE_INODE from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_CACHE_INODE_LRU from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_HASHTABLE from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_HASHTABLE_CACHE from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_DUPREQ from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_INIT from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_MAIN from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_IDMAPPER from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_NFS_READDIR from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_NFS_V4_LOCK from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_CONFIG from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_CLIENTID from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_SESSIONS from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_PNFS from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_RW_LOCK from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_NLM from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_RPC from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_TIRPC from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetNTIRPCLogLevel :LOG :NULL :LOG: Changed RPC_Debug_Flags from 7 to 5
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_NFS_CB from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_THREAD from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_NFS_V4_ACL from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_STATE from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_9P from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_9P_DISPATCH from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_FSAL_UP from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_DBUS from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] SetComponentLogLevel :LOG :NULL :LOG: Changing log level of COMPONENT_NFS_MSK from NIV_EVENT to NIV_WARN
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] nfs_Init_svc :DISP :CRIT :Cannot acquire credentials for principal nfs
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] find_keytab_entry :NFS CB :WARN :Configuration file does not specify default realm while getting default realm name
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] gssd_refresh_krb5_machine_credential :NFS CB :CRIT :ERROR: gssd_refresh_krb5_machine_credential: no usable keytab entry found in keytab /etc/krb5.keytab for connection with host localhost
23/04/2024 15:38:31 : epoch 6627d5f7 : hpe-nfs-a899b66f-2d72-433a-9afb-7bf9e1193358-7fc8654674-vn9bw : ganesha.nfsd-1[main] nfs_rpc_cb_init_ccache :NFS STARTUP :WARN :gssd_refresh_krb5_machine_credential failed (-1765328160:2)

can you look into this and provide a solution?

datamattsson commented 2 months ago

What is the actual problem here? Clients can't connect? It doesn't look fatal and the Pod is running.

nin0-0 commented 2 months ago 
Events:
  Type     Reason                  Age                  From                     Message
  ----     ------                  ----                 ----                     -------
  Normal   Scheduled               17m                  default-scheduler        Successfully assigned tncp/analytics-stream-0 to bl9336
  Normal   SuccessfulAttachVolume  17m                  attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-1911cbd3-8726-460c-b3fa-168e55a2bbb6"
  Normal   SuccessfulAttachVolume  17m                  attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-8b72d900-5050-463b-aaac-034c1c30cbf9"
  Normal   SuccessfulAttachVolume  17m                  attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-38d75969-8769-4f69-8437-877aafa43da7"
  Normal   SuccessfulAttachVolume  17m                  attachdetach-controller  AttachVolume.Attach succeeded for volume "pvc-ff859e43-8713-4efa-bd8d-66947d44df0b"
  Warning  FailedMount             16m                  kubelet                  MountVolume.SetUp failed for volume "pvc-38d75969-8769-4f69-8437-877aafa43da7" : rpc error: code = Internal desc = Error mounting nfs share 172.30.13.196:/export at /var/lib/kubelet/pods/e187cd70-6062-4965-912a-6fb6090ca8dc/volumes/kubernetes.io~csi/pvc-38d75969-8769-4f69-8437-877aafa43da7/mount, err error command mount with pid: 343 killed as timeout of 60 seconds reached
  Warning  FailedMount             15m                  kubelet                  Unable to attach or mount volumes: unmounted volumes=[logs], unattached volumes=[keystore-security sessions-security content-analytics logs kube-api-access-xz2w5]: timed out waiting for the condition
  Warning  FailedMount             15m                  kubelet                  MountVolume.SetUp failed for volume "pvc-38d75969-8769-4f69-8437-877aafa43da7" : rpc error: code = Internal desc = Error mounting nfs share 172.30.13.196:/export at /var/lib/kubelet/pods/e187cd70-6062-4965-912a-6fb6090ca8dc/volumes/kubernetes.io~csi/pvc-38d75969-8769-4f69-8437-877aafa43da7/mount, err error command mount with pid: 364 killed as timeout of 60 seconds reached
  Warning  FailedMount             14m                  kubelet                  MountVolume.SetUp failed for volume "pvc-38d75969-8769-4f69-8437-877aafa43da7" : rpc error: code = Internal desc = Error mounting nfs share 172.30.13.196:/export at /var/lib/kubelet/pods/e187cd70-6062-4965-912a-6fb6090ca8dc/volumes/kubernetes.io~csi/pvc-38d75969-8769-4f69-8437-877aafa43da7/mount, err error command mount with pid: 369 killed as timeout of 60 seconds reached
  Warning  FailedMount             13m                  kubelet                  MountVolume.SetUp failed for volume "pvc-38d75969-8769-4f69-8437-877aafa43da7" : rpc error: code = Internal desc = Error mounting nfs share 172.30.13.196:/export at /var/lib/kubelet/pods/e187cd70-6062-4965-912a-6fb6090ca8dc/volumes/kubernetes.io~csi/pvc-38d75969-8769-4f69-8437-877aafa43da7/mount, err error command mount with pid: 374 killed as timeout of 60 seconds reached
  Warning  FailedMount             12m                  kubelet                  MountVolume.SetUp failed for volume "pvc-38d75969-8769-4f69-8437-877aafa43da7" : rpc error: code = Internal desc = Error mounting nfs share 172.30.13.196:/export at /var/lib/kubelet/pods/e187cd70-6062-4965-912a-6fb6090ca8dc/volumes/kubernetes.io~csi/pvc-38d75969-8769-4f69-8437-877aafa43da7/mount, err error command mount with pid: 379 killed as timeout of 60 seconds reached
  Warning  FailedMount             11m                  kubelet                  Unable to attach or mount volumes: unmounted volumes=[logs], unattached volumes=[sessions-security content-analytics logs kube-api-access-xz2w5 keystore-security]: timed out waiting for the condition
  Warning  FailedMount             11m                  kubelet                  MountVolume.SetUp failed for volume "pvc-38d75969-8769-4f69-8437-877aafa43da7" : rpc error: code = Internal desc = Error mounting nfs share 172.30.13.196:/export at /var/lib/kubelet/pods/e187cd70-6062-4965-912a-6fb6090ca8dc/volumes/kubernetes.io~csi/pvc-38d75969-8769-4f69-8437-877aafa43da7/mount, err error command mount with pid: 384 killed as timeout of 60 seconds reached
  Warning  FailedMount             9m34s (x2 over 13m)  kubelet                  Unable to attach or mount volumes: unmounted volumes=[logs], unattached volumes=[kube-api-access-xz2w5 keystore-security sessions-security content-analytics logs]: timed out waiting for the condition
  Warning  FailedMount             25s (x9 over 10m)    kubelet                  (combined from similar events): MountVolume.SetUp failed for volume "pvc-38d75969-8769-4f69-8437-877aafa43da7" : rpc error: code = Internal desc = Error mounting nfs share 172.30.13.196:/export at /var/lib/kubelet/pods/e187cd70-6062-4965-912a-6fb6090ca8dc/volumes/kubernetes.io~csi/pvc-38d75969-8769-4f69-8437-877aafa43da7/mount, err error command mount with pid: 444 killed as timeout of 60 seconds reached

some pods using RWX have issues but not all

datamattsson commented 2 months ago

The NFS client is most likely unable to reach the NFS Service. Can you file a support case with HPE to have this looked at in detail?

nin0-0 commented 2 months ago

I had opened the case but somehow you are more responsive here :) I posted your comment now