Closed dc232 closed 1 year ago
dc232
commented
2 years ago seems like the most consistent message is
time="2022-02-27T21:24:55Z" level=info msg="CRD not found." CRD=tridentprovisioners.trident.netapp.iocan only assume based on logs that this is bug
balaramesh
commented
2 years ago Hi @dc232
Can you confirm if 10.0.0.68:/pvc-1d9eeaac-e0fe-498c-8376-9ee39c30fb2d is mountable by your Kubernetes cluster?
Since this is with azure-netapp-files, you should check the export policy tab of the associated volume. You will also need to confirm that 10.0.0.68 is accessible by the node in question. This looks to be a n/w issue
dc232
commented
2 years ago Hi @balaramesh sorry for the late reply, i can now confirm that the network issue has been resolved via VNET peering however the exit 32 issue still remains and leaves the PVC in a pending state, via the nodes I am able to do the following
via azue bastion
ubuntu@aks-akscluster-38019429-vmss000000:~$ sudo ufw status
Status: inactive
ubuntu@aks-akscluster-38019429-vmss000000:~$ sudo apt-get install nfs-common
Reading package lists... Done
Building dependency tree
Reading state information... Done
nfs-common is already the newest version (1:1.3.4-2.1ubuntu5.5).
The following packages were automatically installed and are no longer required:
libapr1 libaprutil1
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
mount target IP: 192.1.0.4 --> this is the interface of the netapp appliance which is vnet peered
ubuntu@aks-akscluster-38019429-vmss000000:~$ ping 192.1.0.4
PING 192.1.0.4 (192.1.0.4) 56(84) bytes of data.
64 bytes from 192.1.0.4: icmp_seq=1 ttl=63 time=2.54 ms
64 bytes from 192.1.0.4: icmp_seq=2 ttl=63 time=1.45 ms
64 bytes from 192.1.0.4: icmp_seq=3 ttl=63 time=1.39 ms
64 bytes from 192.1.0.4: icmp_seq=4 ttl=63 time=1.47 ms
64 bytes from 192.1.0.4: icmp_seq=5 ttl=63 time=1.45 ms
64 bytes from 192.1.0.4: icmp_seq=6 ttl=63 time=1.33 ms
64 bytes from 192.1.0.4: icmp_seq=7 ttl=63 time=1.37 ms
64 bytes from 192.1.0.4: icmp_seq=8 ttl=63 time=1.53 ms
we can see the mount ip from the vnet peer, this proves a bug in the CSI or WordPress bitnmi helm char
ubuntu@aks-akscluster-38019429-vmss000000:~$ sudo mount -t nfs -o rw,hard,rsize=65536,wsize=65536,sec=sys,vers=4.1,tcp 192.1.0.4:/pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789 pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789
ubuntu@aks-akscluster-38019429-vmss000000:~$ cd pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789/
-bash: cd: pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789/: Permission denied
ubuntu@aks-akscluster-38019429-vmss000000:~$ sudo cd pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789/
sudo: cd: command not found
above is interesting
ubuntu@aks-akscluster-38019429-vmss000000:~$ ls -l
total 4
drwxrwx--- 2 nobody 4294967294 4096 Mar 13 14:05 pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789
hmm, wonder why the above is creating folders owned by nobody?
ubuntu@aks-akscluster-38019429-vmss000000:~$ less /etc/passwd
ubuntu:x:1000:1000:Ubuntu:/home/ubuntu:/bin/bash
ubuntu@aks-akscluster-38019429-vmss000000:~$ sudo chown ubuntu:ubuntu pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789
ubuntu@aks-akscluster-38019429-vmss000000:~$ cd pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789/
ubuntu@aks-akscluster-38019429-vmss000000:~/pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789$
Remounted as lost connection with azure bastion
ubuntu@aks-akscluster-38019429-vmss000000:~/pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789$ sudo mount -t nfs -o rw,hard,rsize=65536,wsize=65536,sec=sys,vers=4.1,tcp 192.1.0.4:/pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789 pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789
mount.nfs: mount point pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789 does not exist
ubuntu@aks-akscluster-38019429-vmss000000:~/pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789$ cd ..
ubuntu@aks-akscluster-38019429-vmss000000:~$ sudo mount -t nfs -o rw,hard,rsize=65536,wsize=65536,sec=sys,vers=4.1,tcp 192.1.0.4:/pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789 pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789
ubuntu@aks-akscluster-38019429-vmss000000:~$
ubuntu@aks-akscluster-38019429-vmss000000:~$
ubuntu@aks-akscluster-38019429-vmss000000:~$ cd pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789/
ubuntu@aks-akscluster-38019429-vmss000000:~/pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789$ ls -l
total 1074300
-rw-r----- 1 ubuntu ubuntu 1095761920 Dec 21 2012 linuxmint-14-kde-dvd-64bit.iso
ubuntu@aks-akscluster-38019429-vmss000000:~/pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789$ wget https://mirror.math.princeton.edu/pub/ubuntu-iso/20.04.4/ubuntu-20.04.4-desktop-amd64.iso
--2022-03-13 14:58:30-- https://mirror.math.princeton.edu/pub/ubuntu-iso/20.04.4/ubuntu-20.04.4-desktop-amd64.iso
Resolving mirror.math.princeton.edu (mirror.math.princeton.edu)... 128.112.18.21
Connecting to mirror.math.princeton.edu (mirror.math.princeton.edu)|128.112.18.21|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3379068928 (3.1G) [application/octet-stream]
Saving to: ‘ubuntu-20.04.4-desktop-amd64.iso’
ubuntu-20.04.4-desktop-amd64.iso 100%[===================================================================================================================================================================>] 3.15G 5.38MB/s in 6m 9s
2022-03-13 15:07:00 (8.73 MB/s) - ‘ubuntu-20.04.4-desktop-amd64.iso’ saved [3379068928/3379068928]
The above leads me to belive thee is either a bug with he CSI or the bitnami Wordpress helm chart, however i am not sure which one is the culprit
dc232
commented
2 years ago Curent Kubernetes output
kubectl get po -A --kubeconfig config
NAME READY STATUS RESTARTS AGE
phpmyadmin-69bc76d6c4-d94wb 2/2 Running 0 77m
wordpress-6bcfb47ff8-2q74h 0/2 ContainerCreating 0 77m
wordpress-memcached-79545df5d7-w9lj4 1/1 Running 0 77m
netapp trident-csi-558f9448cd-gt7j6 6/6 Running 0 93m
netapp trident-csi-fv8st 2/2 Running 0 77m
netapp trident-csi-hpcfq 2/2 Running 0 93m
netapp trident-operator-5b796496d5-gj97c 1/1 Running 0 93mkubectl get pv,pvc --kubeconfig config
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM
STORAGECLASS REASON AGE
persistentvolume/pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789 100Gi RWX Delete Bound default/wordpress
wordpress-storage-class-netapp 73m
persistentvolume/pvc-25f174fe-47ef-4c53-b023-b3771ccad845 10Gi RWO Delete Bound monitoring/grafana-pvc
default 79m
persistentvolume/pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd 100Gi RWO Delete Bound monitoring/prometheus-prometheus-kube-prometheus-promethe
us-db-prometheus-prometheus-kube-prometheus-prometheus-0 wordpress-storage-class-netapp 76m
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/wordpress Bound pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789 100Gi RWX wordpress-storage-class-netapp 78mkubectl describe po wordpress-6bcfb47ff8-2q74h --kubeconfig config
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedMount 34m (x9 over 65m) kubelet Unable to attach or mount volumes: unmounted volumes=[wordpress-data], unattached volumes=[wordpress-data custom-po
stinit kube-api-access-tf8kh]: timed out waiting for the condition
Warning FailedMount 13m (x8 over 72m) kubelet Unable to attach or mount volumes: unmounted volumes=[wordpress-data], unattached volumes=[custom-postinit kube-api
-access-tf8kh wordpress-data]: timed out waiting for the condition
Warning FailedMount 3m8s (x43 over 74m) kubelet MountVolume.SetUp failed for volume "pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789" : rpc error: code = Internal desc =
error mounting NFS volume 192.1.0.4:/pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789 on mountpoint /var/lib/kubelet/pods/d02f9bbf-8a69-49c1-ba61-9532f0a239c1/volumes/kubernetes
.io~csi/pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789/mount: exit status 32kubectl logs trident-csi-558f9448cd-gt7j6 -n netapp --kubeconfig config -c trident-main
time="2022-03-13T13:48:08Z" level=info msg="Running Trident storage orchestrator." binary=/trident_orchestrator build_time="Mon Jan 31 15:52:06 UTC 2022" version=22.01.0
time="2022-03-13T13:48:08Z" level=info msg="Created Kubernetes clients." namespace=netapp version=v1.23.3
time="2022-03-13T13:48:08Z" level=info msg="Initializing metrics frontend." address=":8001"
time="2022-03-13T13:48:08Z" level=info msg="Added frontend." name=metrics
time="2022-03-13T13:48:08Z" level=info msg="Initializing K8S helper frontend." requestID=5eec05f3-176b-45c2-b4ee-c166eb32ff11 requestSource=Internal
time="2022-03-13T13:48:08Z" level=info msg="K8S helper determined the container orchestrator version." gitVersion=v1.23.3 requestID=5eec05f3-176b-45c2-b4ee-c166eb32ff11
requestSource=Internal version=1.23
time="2022-03-13T13:48:08Z" level=info msg="Added frontend." name=k8s_csi_helper
time="2022-03-13T13:48:08Z" level=info msg="Initializing CSI frontend." name=aks-akscluster-38019429-vmss000000 version=22.01.0
time="2022-03-13T13:48:08Z" level=info msg="Enabling controller service capability." capability=CREATE_DELETE_VOLUME
time="2022-03-13T13:48:08Z" level=info msg="Enabling controller service capability." capability=PUBLISH_UNPUBLISH_VOLUME
time="2022-03-13T13:48:08Z" level=info msg="Enabling controller service capability." capability=LIST_VOLUMES
time="2022-03-13T13:48:08Z" level=info msg="Enabling controller service capability." capability=CREATE_DELETE_SNAPSHOT
time="2022-03-13T13:48:08Z" level=info msg="Enabling controller service capability." capability=LIST_SNAPSHOTS
time="2022-03-13T13:48:08Z" level=info msg="Enabling controller service capability." capability=EXPAND_VOLUME
time="2022-03-13T13:48:08Z" level=info msg="Enabling controller service capability." capability=CLONE_VOLUME
time="2022-03-13T13:48:08Z" level=info msg="Enabling controller service capability." capability=LIST_VOLUMES_PUBLISHED_NODES
time="2022-03-13T13:48:08Z" level=info msg="Enabling volume access mode." mode=SINGLE_NODE_WRITER
time="2022-03-13T13:48:08Z" level=info msg="Enabling volume access mode." mode=SINGLE_NODE_READER_ONLY
time="2022-03-13T13:48:08Z" level=info msg="Enabling volume access mode." mode=MULTI_NODE_READER_ONLY
time="2022-03-13T13:48:08Z" level=info msg="Enabling volume access mode." mode=MULTI_NODE_SINGLE_WRITER
time="2022-03-13T13:48:08Z" level=info msg="Enabling volume access mode." mode=MULTI_NODE_MULTI_WRITER
time="2022-03-13T13:48:08Z" level=info msg="Added frontend." name=csi
time="2022-03-13T13:48:08Z" level=info msg="Initializing Trident CRD controller frontend." namespace=netapp
time="2022-03-13T13:48:08Z" level=info msg="Creating event broadcaster."
time="2022-03-13T13:48:08Z" level=info msg="Setting up CRD controller event handlers."
time="2022-03-13T13:48:08Z" level=info msg="Added frontend." name=crd
time="2022-03-13T13:48:08Z" level=info msg="Initializing HTTP REST frontend." address="127.0.0.1:8000"
time="2022-03-13T13:48:08Z" level=info msg="Added frontend." name="HTTP REST"
time="2022-03-13T13:48:08Z" level=info msg="Initializing HTTPS REST frontend." address=":8443"
time="2022-03-13T13:48:08Z" level=info msg="Added frontend." name="HTTPS REST"
time="2022-03-13T13:48:08Z" level=info msg="Activating metrics frontend." address=":8001"
time="2022-03-13T13:48:08Z" level=info msg="Activating HTTP REST frontend." address="127.0.0.1:8000"
time="2022-03-13T13:48:08Z" level=info msg="Activating HTTPS REST frontend." address=":8443"
time="2022-03-13T13:48:08Z" level=warning msg="Persistent state version not found, creating." OrchestratorAPIVersion=1 PersistentStoreVersion=crdv1 requestID=1f8764fd-9c
f6-4605-961a-beefbe861c51 requestSource=Internal
time="2022-03-13T13:48:09Z" level=info msg="Added 0 existing volume(s)" requestID=1f8764fd-9cf6-4605-961a-beefbe861c51 requestSource=Internal
time="2022-03-13T13:48:09Z" level=info msg="Trident bootstrapped successfully."
time="2022-03-13T13:48:09Z" level=info msg="Activating K8S helper frontend." requestID=1bfa2756-f1bb-4226-b7a8-3451405dcae3 requestSource=Internal
time="2022-03-13T13:48:09Z" level=info msg="Activating CRD frontend."
time="2022-03-13T13:48:09Z" level=info msg="Activating CSI frontend." requestID=f0ff9850-39ce-41d1-971d-7bc6038e6f3a requestSource=Internal
time="2022-03-13T13:48:09Z" level=info msg="Listening for GRPC connections." name=/plugin/csi.sock net=unix
time="2022-03-13T13:48:09Z" level=info msg="Starting Trident CRD controller."
time="2022-03-13T13:48:09Z" level=info msg="Waiting for informer caches to sync."
time="2022-03-13T13:48:09Z" level=info msg="Starting periodic node access reconciliation service." requestID=e4523045-6479-49a9-8e06-2d82b726d607 requestSource=Periodic
I0313 13:48:10.643763 1 request.go:665] Waited for 1.153890579s due to client-side throttling, not priority and fairness, request: GET:https://10.0.0.1:443/apis/tr
ident.netapp.io/v1/namespaces/netapp/tridentversions?limit=500&resourceVersion=0
time="2022-03-13T13:48:11Z" level=info msg="Starting workers."
time="2022-03-13T13:48:11Z" level=info msg="Started workers."
time="2022-03-13T13:48:39Z" level=info msg="Determined topology labels for node: map[topology.kubernetes.io/region:uksouth topology.kubernetes.io/zone:uksouth-1]" node=a
ks-akscluster-38019429-vmss000000 requestID=619cafed-5bd0-46fb-8fd3-988e5ff94ca2 requestSource=REST
time="2022-03-13T13:48:39Z" level=info msg="Added a new node." handler=AddOrUpdateNode node=aks-akscluster-38019429-vmss000000 requestID=619cafed-5bd0-46fb-8fd3-988e5ff9
4ca2 requestSource=REST
time="2022-03-13T13:51:01Z" level=warning msg="The debug setting in the configuration file is now ignored; use the command line --debug switch instead." crdControllerEve
nt=add requestID=e18d2b61-333f-4698-9e76-700660a86bd5 requestSource=CRD
time="2022-03-13T13:51:02Z" level=info msg="Discovered Azure resources." capacityPools=1 crdControllerEvent=add requestID=e18d2b61-333f-4698-9e76-700660a86bd5 requestSou
rce=CRD resourceGroups=1 subnets=1
time="2022-03-13T13:51:02Z" level=info msg="Storage driver initialized." crdControllerEvent=add driver=azure-netapp-files requestID=e18d2b61-333f-4698-9e76-700660a86bd5
requestSource=CRD
time="2022-03-13T13:51:02Z" level=info msg="Created new storage backend." backend="&{0xc0004e23c0 netapp-netapppool true online map[netapp-netapppool_pool:0xc0003b2420]
map[] false}" crdControllerEvent=add requestID=e18d2b61-333f-4698-9e76-700660a86bd5 requestSource=CRD
time="2022-03-13T13:51:02Z" level=info msg="Newly added backend satisfies no storage classes." backend=netapp-netapppool crdControllerEvent=add requestID=e18d2b61-333f-4
698-9e76-700660a86bd5 requestSource=CRD
time="2022-03-13T13:51:02Z" level=info msg="Synced 'netapp/nett-app-backend'" crdControllerEvent=add logSource=trident-crd-controller requestID=e18d2b61-333f-4698-9e76-7
00660a86bd5 requestSource=CRD
time="2022-03-13T13:51:02Z" level=info msg=-------------------------------------------------
time="2022-03-13T13:51:02Z" level=info msg=-------------------------------------------------
time="2022-03-13T13:57:56Z" level=info msg="Storage class satisfied by 1 storage pools." requestID=b5764e05-f67b-4fed-abf0-b00d09acb7cd requestSource=Kubernetes storageC
lass=wordpress-storage-class-netapp
time="2022-03-13T13:57:56Z" level=info msg="K8S helper added a storage class." name=wordpress-storage-class-netapp parameters="map[backendType:azure-netapp-files fsType:
nfs]" provisioner=csi.trident.netapp.io requestID=b5764e05-f67b-4fed-abf0-b00d09acb7cd requestSource=Kubernetes
time="2022-03-13T13:58:22Z" level=info msg="Found PVC for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." UID=b399eae4-6210-45b9-95f0-a2fbbb2223dd name=prome
theus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0 namespace=monitoring requestID=f9cf3f03-383d-45d7-9a1c-8c63b14c08f4 req
uestSource=CSI size="{{107374182400 0} {<nil>} 100Gi BinarySI}" storageClass=wordpress-storage-class-netapp
time="2022-03-13T13:58:22Z" level=info msg="Found storage class for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." name=wordpress-storage-class-netapp reque
stID=f9cf3f03-383d-45d7-9a1c-8c63b14c08f4 requestSource=CSI
time="2022-03-13T13:58:24Z" level=info msg="Volume created." API=VolumesClient.CreateOrUpdate correlationID=0b666528-c616-4b2c-b3b4-e1250c647475 creationToken=pvc-b399ea
e4-6210-45b9-95f0-a2fbbb2223dd requestID=f9cf3f03-383d-45d7-9a1c-8c63b14c08f4 requestSource=CSI volume=netapp_rg/netapp-netappaccount/netapp-netapppool/pvc-b399eae4-6210
-45b9-95f0-a2fbbb2223dd
time="2022-03-13T13:58:24Z" level=info msg="Waiting for volume state." desiredState=Succeeded requestID=f9cf3f03-383d-45d7-9a1c-8c63b14c08f4 requestSource=CSI
time="2022-03-13T13:58:32Z" level=error msg="Volume state was not Succeeded after 10.00 seconds." requestID=f9cf3f03-383d-45d7-9a1c-8c63b14c08f4 requestSource=CSI
time="2022-03-13T13:58:32Z" level=warning msg="Volume still creating on this backend." backend=netapp-netapppool backendUUID=5b791bfd-98e7-4983-a00f-77d57d2cb3b8 error="
volume state is Creating, not Succeeded" pool=netapp-netapppool_pool requestID=f9cf3f03-383d-45d7-9a1c-8c63b14c08f4 requestSource=CSI volume=pvc-b399eae4-6210-45b9-95f0-
a2fbbb2223dd
time="2022-03-13T13:58:32Z" level=error msg="GRPC error: rpc error: code = Unknown desc = volume state is Creating, not Succeeded" requestID=f9cf3f03-383d-45d7-9a1c-8c63
b14c08f4 requestSource=CSI
time="2022-03-13T13:58:40Z" level=info msg="Found PVC for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." UID=b399eae4-6210-45b9-95f0-a2fbbb2223dd name=prome
theus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0 namespace=monitoring requestID=33aca4d8-5585-47b0-aa6a-b20ef073c061 req
uestSource=CSI size="{{107374182400 0} {<nil>} 100Gi BinarySI}" storageClass=wordpress-storage-class-netapp
time="2022-03-13T13:58:40Z" level=info msg="Found storage class for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." name=wordpress-storage-class-netapp reque
stID=33aca4d8-5585-47b0-aa6a-b20ef073c061 requestSource=CSI
time="2022-03-13T13:58:40Z" level=warning msg="Volume still creating on this backend." backend=netapp-netapppool backendUUID=5b791bfd-98e7-4983-a00f-77d57d2cb3b8 error="
volume state is still Creating, not Succeeded" pool=netapp-netapppool_pool requestID=33aca4d8-5585-47b0-aa6a-b20ef073c061 requestSource=CSI volume=pvc-b399eae4-6210-45b9
-95f0-a2fbbb2223dd
time="2022-03-13T13:58:40Z" level=error msg="GRPC error: rpc error: code = Unknown desc = volume state is still Creating, not Succeeded" requestID=33aca4d8-5585-47b0-aa6
a-b20ef073c061 requestSource=CSI
time="2022-03-13T13:58:56Z" level=info msg="Found PVC for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." UID=b399eae4-6210-45b9-95f0-a2fbbb2223dd name=prome
theus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0 namespace=monitoring requestID=07cd36e7-f7a9-4893-a060-43a7663604fc req
uestSource=CSI size="{{107374182400 0} {<nil>} 100Gi BinarySI}" storageClass=wordpress-storage-class-netapp
time="2022-03-13T13:58:56Z" level=info msg="Found storage class for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." name=wordpress-storage-class-netapp reque
stID=07cd36e7-f7a9-4893-a060-43a7663604fc requestSource=CSI
time="2022-03-13T13:58:56Z" level=warning msg="Volume still creating on this backend." backend=netapp-netapppool backendUUID=5b791bfd-98e7-4983-a00f-77d57d2cb3b8 error="
volume state is still Creating, not Succeeded" pool=netapp-netapppool_pool requestID=07cd36e7-f7a9-4893-a060-43a7663604fc requestSource=CSI volume=pvc-b399eae4-6210-45b9
-95f0-a2fbbb2223dd
time="2022-03-13T13:58:56Z" level=error msg="GRPC error: rpc error: code = Unknown desc = volume state is still Creating, not Succeeded" requestID=07cd36e7-f7a9-4893-a06
0-43a7663604fc requestSource=CSI
time="2022-03-13T13:59:26Z" level=info msg="Found PVC for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." UID=b399eae4-6210-45b9-95f0-a2fbbb2223dd name=prome
theus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0 namespace=monitoring requestID=af88008d-570e-4fe1-bc06-5647849fbb16 req
uestSource=CSI size="{{107374182400 0} {<nil>} 100Gi BinarySI}" storageClass=wordpress-storage-class-netapp
time="2022-03-13T13:59:26Z" level=info msg="Found storage class for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." name=wordpress-storage-class-netapp reque
stID=af88008d-570e-4fe1-bc06-5647849fbb16 requestSource=CSI
time="2022-03-13T13:59:26Z" level=warning msg="Volume still creating on this backend." backend=netapp-netapppool backendUUID=5b791bfd-98e7-4983-a00f-77d57d2cb3b8 error="
volume state is still Creating, not Succeeded" pool=netapp-netapppool_pool requestID=af88008d-570e-4fe1-bc06-5647849fbb16 requestSource=CSI volume=pvc-b399eae4-6210-45b9
-95f0-a2fbbb2223dd
time="2022-03-13T13:59:26Z" level=error msg="GRPC error: rpc error: code = Unknown desc = volume state is still Creating, not Succeeded" requestID=af88008d-570e-4fe1-bc0
6-5647849fbb16 requestSource=CSI
time="2022-03-13T13:59:56Z" level=info msg="Found PVC for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." UID=b399eae4-6210-45b9-95f0-a2fbbb2223dd name=prome
theus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0 namespace=monitoring requestID=e00c1768-17f0-46b0-8ea7-0ee3eea40bfe req
uestSource=CSI size="{{107374182400 0} {<nil>} 100Gi BinarySI}" storageClass=wordpress-storage-class-netapp
time="2022-03-13T13:59:56Z" level=info msg="Found storage class for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." name=wordpress-storage-class-netapp reque
stID=e00c1768-17f0-46b0-8ea7-0ee3eea40bfe requestSource=CSI
time="2022-03-13T13:59:57Z" level=warning msg="Volume still creating on this backend." backend=netapp-netapppool backendUUID=5b791bfd-98e7-4983-a00f-77d57d2cb3b8 error="
volume state is still Creating, not Succeeded" pool=netapp-netapppool_pool requestID=e00c1768-17f0-46b0-8ea7-0ee3eea40bfe requestSource=CSI volume=pvc-b399eae4-6210-45b9
-95f0-a2fbbb2223dd
time="2022-03-13T13:59:57Z" level=error msg="GRPC error: rpc error: code = Unknown desc = volume state is still Creating, not Succeeded" requestID=e00c1768-17f0-46b0-8ea
7-0ee3eea40bfe requestSource=CSI
time="2022-03-13T14:00:27Z" level=info msg="Found PVC for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." UID=b399eae4-6210-45b9-95f0-a2fbbb2223dd name=prome
theus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0 namespace=monitoring requestID=b989e40c-fd15-4c86-9d9e-e5f321cccf7b req
uestSource=CSI size="{{107374182400 0} {<nil>} 100Gi BinarySI}" storageClass=wordpress-storage-class-netapp
time="2022-03-13T14:00:27Z" level=info msg="Found storage class for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." name=wordpress-storage-class-netapp reque
stID=b989e40c-fd15-4c86-9d9e-e5f321cccf7b requestSource=CSI
time="2022-03-13T14:00:27Z" level=warning msg="Volume still creating on this backend." backend=netapp-netapppool backendUUID=5b791bfd-98e7-4983-a00f-77d57d2cb3b8 error="
volume state is still Creating, not Succeeded" pool=netapp-netapppool_pool requestID=b989e40c-fd15-4c86-9d9e-e5f321cccf7b requestSource=CSI volume=pvc-b399eae4-6210-45b9
-95f0-a2fbbb2223dd
time="2022-03-13T14:00:27Z" level=error msg="GRPC error: rpc error: code = Unknown desc = volume state is still Creating, not Succeeded" requestID=b989e40c-fd15-4c86-9d9
e-e5f321cccf7b requestSource=CSI
time="2022-03-13T14:00:57Z" level=info msg="Found PVC for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." UID=b399eae4-6210-45b9-95f0-a2fbbb2223dd name=prome
theus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0 namespace=monitoring requestID=73b66a4d-5cf7-4a86-82d2-35d8b18fa6d5 req
uestSource=CSI size="{{107374182400 0} {<nil>} 100Gi BinarySI}" storageClass=wordpress-storage-class-netapp
time="2022-03-13T14:00:57Z" level=info msg="Found storage class for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." name=wordpress-storage-class-netapp reque
stID=73b66a4d-5cf7-4a86-82d2-35d8b18fa6d5 requestSource=CSI
time="2022-03-13T14:00:57Z" level=warning msg="Volume still creating on this backend." backend=netapp-netapppool backendUUID=5b791bfd-98e7-4983-a00f-77d57d2cb3b8 error="
volume state is still Creating, not Succeeded" pool=netapp-netapppool_pool requestID=73b66a4d-5cf7-4a86-82d2-35d8b18fa6d5 requestSource=CSI volume=pvc-b399eae4-6210-45b9
-95f0-a2fbbb2223dd
time="2022-03-13T14:00:57Z" level=error msg="GRPC error: rpc error: code = Unknown desc = volume state is still Creating, not Succeeded" requestID=73b66a4d-5cf7-4a86-82d
2-35d8b18fa6d5 requestSource=CSI
time="2022-03-13T14:01:27Z" level=info msg="Found PVC for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." UID=b399eae4-6210-45b9-95f0-a2fbbb2223dd name=prome
theus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0 namespace=monitoring requestID=38f60596-12b3-4b93-a066-2ec9351267d5 req
uestSource=CSI size="{{107374182400 0} {<nil>} 100Gi BinarySI}" storageClass=wordpress-storage-class-netapp
time="2022-03-13T14:01:27Z" level=info msg="Found storage class for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." name=wordpress-storage-class-netapp reque
stID=38f60596-12b3-4b93-a066-2ec9351267d5 requestSource=CSI
time="2022-03-13T14:01:27Z" level=info msg="Discovered Azure resources." capacityPools=1 requestID=38f60596-12b3-4b93-a066-2ec9351267d5 requestSource=CSI resourceGroups=
1 subnets=1
time="2022-03-13T14:01:27Z" level=warning msg="Volume still creating on this backend." backend=netapp-netapppool backendUUID=5b791bfd-98e7-4983-a00f-77d57d2cb3b8 error="
volume state is still Creating, not Succeeded" pool=netapp-netapppool_pool requestID=38f60596-12b3-4b93-a066-2ec9351267d5 requestSource=CSI volume=pvc-b399eae4-6210-45b9
-95f0-a2fbbb2223dd
time="2022-03-13T14:01:27Z" level=error msg="GRPC error: rpc error: code = Unknown desc = volume state is still Creating, not Succeeded" requestID=38f60596-12b3-4b93-a06
6-2ec9351267d5 requestSource=CSI
time="2022-03-13T14:01:57Z" level=info msg="Found PVC for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." UID=b399eae4-6210-45b9-95f0-a2fbbb2223dd name=prome
theus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0 namespace=monitoring requestID=276b1543-332c-4be9-8eef-ca70649bd02b req
uestSource=CSI size="{{107374182400 0} {<nil>} 100Gi BinarySI}" storageClass=wordpress-storage-class-netapp
time="2022-03-13T14:01:57Z" level=info msg="Found storage class for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." name=wordpress-storage-class-netapp reque
stID=276b1543-332c-4be9-8eef-ca70649bd02b requestSource=CSI
time="2022-03-13T14:01:57Z" level=warning msg="Volume still creating on this backend." backend=netapp-netapppool backendUUID=5b791bfd-98e7-4983-a00f-77d57d2cb3b8 error="
volume state is still Creating, not Succeeded" pool=netapp-netapppool_pool requestID=276b1543-332c-4be9-8eef-ca70649bd02b requestSource=CSI volume=pvc-b399eae4-6210-45b9
-95f0-a2fbbb2223dd
time="2022-03-13T14:01:57Z" level=error msg="GRPC error: rpc error: code = Unknown desc = volume state is still Creating, not Succeeded" requestID=276b1543-332c-4be9-8ee
f-ca70649bd02b requestSource=CSI
time="2022-03-13T14:02:27Z" level=info msg="Found PVC for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." UID=b399eae4-6210-45b9-95f0-a2fbbb2223dd name=prome
theus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0 namespace=monitoring requestID=7f8234b3-0c46-4804-89ab-edd7d31e110e req
uestSource=CSI size="{{107374182400 0} {<nil>} 100Gi BinarySI}" storageClass=wordpress-storage-class-netapp
time="2022-03-13T14:02:27Z" level=info msg="Found storage class for requested volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd." name=wordpress-storage-class-netapp reque
stID=7f8234b3-0c46-4804-89ab-edd7d31e110e requestSource=CSI
time="2022-03-13T14:02:28Z" level=warning msg="Volume already exists." name=pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd requestID=7f8234b3-0c46-4804-89ab-edd7d31e110e reque
stSource=CSI state=Succeeded
time="2022-03-13T14:02:28Z" level=warning msg="Volume already exists." backend=netapp-netapppool requestID=7f8234b3-0c46-4804-89ab-edd7d31e110e requestSource=CSI volume=
pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd
time="2022-03-13T14:05:13Z" level=info msg="Determined topology labels for node: map[topology.kubernetes.io/region:uksouth topology.kubernetes.io/zone:uksouth-1]" node=a
ks-akscluster-38019429-vmss000001 requestID=fc4ee0c3-e632-4074-8b2f-d9f469446e8e requestSource=REST
time="2022-03-13T14:05:13Z" level=info msg="Added a new node." handler=AddOrUpdateNode node=aks-akscluster-38019429-vmss000001 requestID=fc4ee0c3-e632-4074-8b2f-d9f46944
6e8e requestSource=REST
time="2022-03-13T14:05:18Z" level=info msg="Found PVC for requested volume pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789." UID=0099b9fa-704b-4110-b1c7-cf8dfe2c4789 name=wordp
ress namespace=default requestID=ef959676-4251-4808-b5e1-b84bb1a7d2b8 requestSource=CSI size="{{107374182400 0} {<nil>} 100Gi BinarySI}" storageClass=wordpress-storage-c
lass-netapp
time="2022-03-13T14:05:18Z" level=info msg="Found storage class for requested volume pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789." name=wordpress-storage-class-netapp reque
stID=ef959676-4251-4808-b5e1-b84bb1a7d2b8 requestSource=CSI
time="2022-03-13T14:05:19Z" level=info msg="Volume created." API=VolumesClient.CreateOrUpdate correlationID=91f801e7-9736-460b-aaf0-ac38d84528cf creationToken=pvc-0099b9
fa-704b-4110-b1c7-cf8dfe2c4789 requestID=ef959676-4251-4808-b5e1-b84bb1a7d2b8 requestSource=CSI volume=netapp_rg/netapp-netappaccount/netapp-netapppool/pvc-0099b9fa-704b
-4110-b1c7-cf8dfe2c4789
time="2022-03-13T14:05:19Z" level=info msg="Waiting for volume state." desiredState=Succeeded requestID=ef959676-4251-4808-b5e1-b84bb1a7d2b8 requestSource=CSI
time="2022-03-13T14:05:27Z" level=error msg="Volume state was not Succeeded after 10.00 seconds." requestID=ef959676-4251-4808-b5e1-b84bb1a7d2b8 requestSource=CSI
time="2022-03-13T14:05:27Z" level=warning msg="Volume still creating on this backend." backend=netapp-netapppool backendUUID=5b791bfd-98e7-4983-a00f-77d57d2cb3b8 error="
volume state is Creating, not Succeeded" pool=netapp-netapppool_pool requestID=ef959676-4251-4808-b5e1-b84bb1a7d2b8 requestSource=CSI volume=pvc-0099b9fa-704b-4110-b1c7-
cf8dfe2c4789
time="2022-03-13T14:05:27Z" level=error msg="GRPC error: rpc error: code = Unknown desc = volume state is Creating, not Succeeded" requestID=ef959676-4251-4808-b5e1-b84b
b1a7d2b8 requestSource=CSI
time="2022-03-13T14:05:35Z" level=info msg="Found PVC for requested volume pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789." UID=0099b9fa-704b-4110-b1c7-cf8dfe2c4789 name=wordp
ress namespace=default requestID=b27e72c7-0cc5-4222-ac2d-8654d0dc5bf7 requestSource=CSI size="{{107374182400 0} {<nil>} 100Gi BinarySI}" storageClass=wordpress-storage-c
lass-netapp
time="2022-03-13T14:05:35Z" level=info msg="Found storage class for requested volume pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789." name=wordpress-storage-class-netapp reque
stID=b27e72c7-0cc5-4222-ac2d-8654d0dc5bf7 requestSource=CSI
time="2022-03-13T14:05:35Z" level=warning msg="Volume already exists." name=pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789 requestID=b27e72c7-0cc5-4222-ac2d-8654d0dc5bf7 reque
stSource=CSI state=Succeeded
time="2022-03-13T14:05:35Z" level=warning msg="Volume already exists." backend=netapp-netapppool requestID=b27e72c7-0cc5-4222-ac2d-8654d0dc5bf7 requestSource=CSI volume=
pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789kubectl logs trident-csi-558f9448cd-gt7j6 -n netapp --kubeconfig config -c csi-provisioner
W0313 13:48:16.466379 1 feature_gate.go:237] Setting GA feature gate Topology=true. It will be removed in a future release.
I0313 13:48:16.466415 1 feature_gate.go:245] feature gates: &{map[Topology:true]}
I0313 13:48:16.466425 1 csi-provisioner.go:139] Version: v3.1.0
I0313 13:48:16.466430 1 csi-provisioner.go:162] Building kube configs for running in cluster...
I0313 13:48:16.467747 1 common.go:111] Probing CSI driver for readiness
I0313 13:48:16.470794 1 csi-provisioner.go:206] Detected CSI driver csi.trident.netapp.io
I0313 13:48:16.471942 1 csi-provisioner.go:275] CSI driver supports PUBLISH_UNPUBLISH_VOLUME, watching VolumeAttachments
I0313 13:48:16.472273 1 controller.go:732] Using saving PVs to API server in background
I0313 13:48:16.572825 1 controller.go:811] Starting provisioner controller csi.trident.netapp.io_trident-csi-558f9448cd-gt7j6_b2fb0794-3d1f-4675-8e01-899c5817de0b!
I0313 13:48:16.572930 1 clone_controller.go:66] Starting CloningProtection controller
I0313 13:48:16.573016 1 clone_controller.go:82] Started CloningProtection controller
I0313 13:48:16.573155 1 volume_store.go:97] Starting save volume queue
I0313 13:48:16.674025 1 controller.go:860] Started provisioner controller csi.trident.netapp.io_trident-csi-558f9448cd-gt7j6_b2fb0794-3d1f-4675-8e01-899c5817de0b!
I0313 13:58:22.656495 1 controller.go:1337] provision "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometh
eus-0" class "wordpress-storage-class-netapp": started
W0313 13:58:22.656577 1 controller.go:579] "fstype" is deprecated and will be removed in a future release, please use "csi.storage.k8s.io/fstype" instead
I0313 13:58:22.657145 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prome
theus-kube-prometheus-prometheus-0"
I0313 13:58:32.453817 1 controller.go:1075] Final error received, removing PVC b399eae4-6210-45b9-95f0-a2fbbb2223dd from claims in progress
W0313 13:58:32.453840 1 controller.go:934] Retrying syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd", failure 0
E0313 13:58:32.453868 1 controller.go:957] error syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd": failed to provision volume with StorageClass "wordpress-stor
age-class-netapp": rpc error: code = Unknown desc = volume state is Creating, not Succeeded
I0313 13:58:32.453886 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "wordpress-storage-class-netapp": rpc error: code = Unknown desc = volume state is
Creating, not Succeeded
I0313 13:58:40.454126 1 controller.go:1337] provision "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometh
eus-0" class "wordpress-storage-class-netapp": started
W0313 13:58:40.454203 1 controller.go:579] "fstype" is deprecated and will be removed in a future release, please use "csi.storage.k8s.io/fstype" instead
I0313 13:58:40.454725 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prome
theus-kube-prometheus-prometheus-0"
I0313 13:58:40.577047 1 controller.go:1075] Final error received, removing PVC b399eae4-6210-45b9-95f0-a2fbbb2223dd from claims in progress
W0313 13:58:40.577289 1 controller.go:934] Retrying syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd", failure 1
E0313 13:58:40.577316 1 controller.go:957] error syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd": failed to provision volume with StorageClass "wordpress-stor
age-class-netapp": rpc error: code = Unknown desc = volume state is still Creating, not Succeeded
I0313 13:58:40.577582 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "wordpress-storage-class-netapp": rpc error: code = Unknown desc = volume state is
still Creating, not Succeeded
I0313 13:58:56.577576 1 controller.go:1337] provision "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometh
eus-0" class "wordpress-storage-class-netapp": started
W0313 13:58:56.577644 1 controller.go:579] "fstype" is deprecated and will be removed in a future release, please use "csi.storage.k8s.io/fstype" instead
I0313 13:58:56.577767 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prome
theus-kube-prometheus-prometheus-0"
I0313 13:58:56.669190 1 controller.go:1075] Final error received, removing PVC b399eae4-6210-45b9-95f0-a2fbbb2223dd from claims in progress
W0313 13:58:56.669221 1 controller.go:934] Retrying syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd", failure 2
E0313 13:58:56.669315 1 controller.go:957] error syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd": failed to provision volume with StorageClass "wordpress-stor
age-class-netapp": rpc error: code = Unknown desc = volume state is still Creating, not Succeeded
I0313 13:58:56.669387 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "wordpress-storage-class-netapp": rpc error: code = Unknown desc = volume state is
still Creating, not Succeeded
I0313 13:59:26.669806 1 controller.go:1337] provision "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometh
eus-0" class "wordpress-storage-class-netapp": started
W0313 13:59:26.669896 1 controller.go:579] "fstype" is deprecated and will be removed in a future release, please use "csi.storage.k8s.io/fstype" instead
I0313 13:59:26.670406 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prome
theus-kube-prometheus-prometheus-0"
I0313 13:59:26.751062 1 controller.go:1075] Final error received, removing PVC b399eae4-6210-45b9-95f0-a2fbbb2223dd from claims in progress
W0313 13:59:26.751083 1 controller.go:934] Retrying syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd", failure 3
E0313 13:59:26.751107 1 controller.go:957] error syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd": failed to provision volume with StorageClass "wordpress-stor
age-class-netapp": rpc error: code = Unknown desc = volume state is still Creating, not Succeeded
I0313 13:59:26.751389 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "wordpress-storage-class-netapp": rpc error: code = Unknown desc = volume state is
still Creating, not Succeeded
I0313 13:59:56.752135 1 controller.go:1337] provision "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometh
eus-0" class "wordpress-storage-class-netapp": started
W0313 13:59:56.752205 1 controller.go:579] "fstype" is deprecated and will be removed in a future release, please use "csi.storage.k8s.io/fstype" instead
I0313 13:59:56.752607 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prome
theus-kube-prometheus-prometheus-0"
I0313 13:59:57.260992 1 controller.go:1075] Final error received, removing PVC b399eae4-6210-45b9-95f0-a2fbbb2223dd from claims in progress
W0313 13:59:57.261012 1 controller.go:934] Retrying syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd", failure 4
E0313 13:59:57.261031 1 controller.go:957] error syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd": failed to provision volume with StorageClass "wordpress-stor
age-class-netapp": rpc error: code = Unknown desc = volume state is still Creating, not Succeeded
I0313 13:59:57.261368 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "wordpress-storage-class-netapp": rpc error: code = Unknown desc = volume state is
still Creating, not Succeeded
I0313 14:00:27.261485 1 controller.go:1337] provision "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometh
eus-0" class "wordpress-storage-class-netapp": started
W0313 14:00:27.261560 1 controller.go:579] "fstype" is deprecated and will be removed in a future release, please use "csi.storage.k8s.io/fstype" instead
I0313 14:00:27.262179 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prome
theus-kube-prometheus-prometheus-0"
I0313 14:00:27.390891 1 controller.go:1075] Final error received, removing PVC b399eae4-6210-45b9-95f0-a2fbbb2223dd from claims in progress
W0313 14:00:27.390926 1 controller.go:934] Retrying syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd", failure 5
E0313 14:00:27.390947 1 controller.go:957] error syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd": failed to provision volume with StorageClass "wordpress-stor
age-class-netapp": rpc error: code = Unknown desc = volume state is still Creating, not Succeeded
I0313 14:00:27.390971 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "wordpress-storage-class-netapp": rpc error: code = Unknown desc = volume state is
still Creating, not Succeeded
I0313 14:00:57.391081 1 controller.go:1337] provision "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometh
eus-0" class "wordpress-storage-class-netapp": started
W0313 14:00:57.391169 1 controller.go:579] "fstype" is deprecated and will be removed in a future release, please use "csi.storage.k8s.io/fstype" instead
I0313 14:00:57.391372 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prome
theus-kube-prometheus-prometheus-0"
I0313 14:00:57.480296 1 controller.go:1075] Final error received, removing PVC b399eae4-6210-45b9-95f0-a2fbbb2223dd from claims in progress
W0313 14:00:57.480311 1 controller.go:934] Retrying syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd", failure 6
E0313 14:00:57.480326 1 controller.go:957] error syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd": failed to provision volume with StorageClass "wordpress-stor
age-class-netapp": rpc error: code = Unknown desc = volume state is still Creating, not Succeeded
I0313 14:00:57.480600 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "wordpress-storage-class-netapp": rpc error: code = Unknown desc = volume state is
still Creating, not Succeeded
I0313 14:01:27.480879 1 controller.go:1337] provision "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometh
eus-0" class "wordpress-storage-class-netapp": started
W0313 14:01:27.480953 1 controller.go:579] "fstype" is deprecated and will be removed in a future release, please use "csi.storage.k8s.io/fstype" instead
I0313 14:01:27.481002 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prome
theus-kube-prometheus-prometheus-0"
I0313 14:01:27.796824 1 controller.go:1075] Final error received, removing PVC b399eae4-6210-45b9-95f0-a2fbbb2223dd from claims in progress
W0313 14:01:27.796847 1 controller.go:934] Retrying syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd", failure 7
E0313 14:01:27.796867 1 controller.go:957] error syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd": failed to provision volume with StorageClass "wordpress-stor
age-class-netapp": rpc error: code = Unknown desc = volume state is still Creating, not Succeeded
I0313 14:01:27.797064 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "wordpress-storage-class-netapp": rpc error: code = Unknown desc = volume state is
still Creating, not Succeeded
I0313 14:01:57.797001 1 controller.go:1337] provision "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometh
eus-0" class "wordpress-storage-class-netapp": started
W0313 14:01:57.797071 1 controller.go:579] "fstype" is deprecated and will be removed in a future release, please use "csi.storage.k8s.io/fstype" instead
I0313 14:01:57.797570 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prome
theus-kube-prometheus-prometheus-0"
I0313 14:01:57.937150 1 controller.go:1075] Final error received, removing PVC b399eae4-6210-45b9-95f0-a2fbbb2223dd from claims in progress
W0313 14:01:57.937586 1 controller.go:934] Retrying syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd", failure 8
E0313 14:01:57.937613 1 controller.go:957] error syncing claim "b399eae4-6210-45b9-95f0-a2fbbb2223dd": failed to provision volume with StorageClass "wordpress-stor
age-class-netapp": rpc error: code = Unknown desc = volume state is still Creating, not Succeeded
I0313 14:01:57.937936 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "wordpress-storage-class-netapp": rpc error: code = Unknown desc = volume state is
still Creating, not Succeeded
I0313 14:02:27.938371 1 controller.go:1337] provision "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometh
eus-0" class "wordpress-storage-class-netapp": started
W0313 14:02:27.938447 1 controller.go:579] "fstype" is deprecated and will be removed in a future release, please use "csi.storage.k8s.io/fstype" instead
I0313 14:02:27.938664 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prome
theus-kube-prometheus-prometheus-0"
I0313 14:02:28.185989 1 controller.go:858] successfully created PV pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd for PVC prometheus-prometheus-kube-prometheus-prometheu
s-db-prometheus-prometheus-kube-prometheus-prometheus-0 and csi volume name pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd
I0313 14:02:28.186028 1 controller.go:1442] provision "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometh
eus-0" class "wordpress-storage-class-netapp": volume "pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd" provisioned
I0313 14:02:28.186041 1 controller.go:1455] provision "monitoring/prometheus-prometheus-kube-prometheus-prometheus-db-prometheus-prometheus-kube-prometheus-prometh
eus-0" class "wordpress-storage-class-netapp": succeeded
I0313 14:02:28.193407 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"monitoring", Name:"prometheus-prometheus-kube-prometheus-pr
ometheus-db-prometheus-prometheus-kube-prometheus-prometheus-0", UID:"b399eae4-6210-45b9-95f0-a2fbbb2223dd", APIVersion:"v1", ResourceVersion:"7038", FieldPath:""}): typ
e: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-b399eae4-6210-45b9-95f0-a2fbbb2223dd
I0313 14:05:18.404717 1 controller.go:1337] provision "default/wordpress" class "wordpress-storage-class-netapp": started
W0313 14:05:18.405162 1 controller.go:579] "fstype" is deprecated and will be removed in a future release, please use "csi.storage.k8s.io/fstype" instead
I0313 14:05:18.405523 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"wordpress", UID:"0099b9fa-704b-4110-b1c7-cf
8dfe2c4789", APIVersion:"v1", ResourceVersion:"10033", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "defau
lt/wordpress"
I0313 14:05:27.442119 1 controller.go:1075] Final error received, removing PVC 0099b9fa-704b-4110-b1c7-cf8dfe2c4789 from claims in progress
W0313 14:05:27.442147 1 controller.go:934] Retrying syncing claim "0099b9fa-704b-4110-b1c7-cf8dfe2c4789", failure 0
E0313 14:05:27.442166 1 controller.go:957] error syncing claim "0099b9fa-704b-4110-b1c7-cf8dfe2c4789": failed to provision volume with StorageClass "wordpress-stor
age-class-netapp": rpc error: code = Unknown desc = volume state is Creating, not Succeeded
I0313 14:05:27.442423 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"wordpress", UID:"0099b9fa-704b-4110-b1c7-cf
8dfe2c4789", APIVersion:"v1", ResourceVersion:"10033", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "wordpre
ss-storage-class-netapp": rpc error: code = Unknown desc = volume state is Creating, not Succeeded
I0313 14:05:35.442355 1 controller.go:1337] provision "default/wordpress" class "wordpress-storage-class-netapp": started
W0313 14:05:35.442457 1 controller.go:579] "fstype" is deprecated and will be removed in a future release, please use "csi.storage.k8s.io/fstype" instead
I0313 14:05:35.442712 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"wordpress", UID:"0099b9fa-704b-4110-b1c7-cf
8dfe2c4789", APIVersion:"v1", ResourceVersion:"10033", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "defau
lt/wordpress"
I0313 14:05:35.638854 1 controller.go:858] successfully created PV pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789 for PVC wordpress and csi volume name pvc-0099b9fa-704b
-4110-b1c7-cf8dfe2c4789
I0313 14:05:35.639292 1 controller.go:1442] provision "default/wordpress" class "wordpress-storage-class-netapp": volume "pvc-0099b9fa-704b-4110-b1c7-cf8dfe2c4789"
provisioned
I0313 14:05:35.639472 1 controller.go:1455] provision "default/wordpress" class "wordpress-storage-class-netapp": succeeded
I0313 14:05:35.645968 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"wordpress", UID:"0099b9fa-704b-4110-b1c7-cf
8dfe2c4789", APIVersion:"v1", ResourceVersion:"10033", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-0099b9fa-704b-4
110-b1c7-cf8dfe2c4789PS C:\Users\danny\Desktop> kubectl logs trident-csi-558f9448cd-gt7j6 -n netapp --kubeconfig config -c csi-attacher
I0313 13:48:19.167837 1 main.go:92] Version: v3.4.0
I0313 13:48:19.169447 1 common.go:111] Probing CSI driver for readiness
I0313 13:48:19.172137 1 main.go:148] CSI driver name: "csi.trident.netapp.io"
I0313 13:48:19.172899 1 main.go:223] CSI driver supports ControllerPublishUnpublish, using real CSI handler
I0313 13:48:19.172920 1 main.go:231] CSI driver supports list volumes published nodes. Using capability to reconcile volume attachment objects with actual backend
state
I0313 13:48:19.173005 1 controller.go:128] Starting CSI attacher
I0313 14:02:28.690526 1 csi_handler.go:248] Attaching "csi-e885b2bab91f5ca1dd74ec1802fa49e42d68d428fe971bf19784dae3dd9fff48"
I0313 14:02:28.989566 1 csi_handler.go:261] Attached "csi-e885b2bab91f5ca1dd74ec1802fa49e42d68d428fe971bf19784dae3dd9fff48"
I0313 14:05:36.502663 1 csi_handler.go:248] Attaching "csi-5c376092415886c47032b0ba380374146edfe61d3e5beefd99cfc6218531c07d"
I0313 14:05:36.703642 1 csi_handler.go:261] Attached "csi-5c376092415886c47032b0ba380374146edfe61d3e5beefd99cfc6218531c07d"@balaramesh by export policy do you mean
Defaults:
Export Rule: 0.0.0.0/0?
dc232
commented
2 years ago @balaramesh where can I find information about the export policy for azure? this could aid in troubleshooting the error, from the logs it looks like the volume is trying to be created in Netapp but failing for some reason to be created within a 10-second time frame, is there a way this time frame can be increased?
I'm using a contributor role or the secret
also is there any documentation that aid in muting the warning
fstype" is deprecated and will be removed in a future release, please use "csi.storage.k8s.io/fstype" insteadhowever stillspeificy NFS4.1 to be used as the protocol?
balaramesh
commented
2 years ago @dc232 if that's the output of the export policy then it looks ok to me. I would recommend opening a ticket with NetApp Support if this issue still persists. You could look at using mountOptions in your storageClass and set vers=4.1 to set the protocol type.
dc232
commented
2 years ago @balaramesh can confirm that is definitely the export policy
I did notice something very odd about about the created volumes but not sure why this is, which seems to be no valid mount target, i'm not sure what's causing this, as NSG rule for peered networks seem to allow for NFS on 111 and 2049 via AllowVnetOutBound and AllowVnetInBound
mount options seem to be set via on the backend config yaml
nfsMountOptions: nfsvers=4would this be the correct for the mountoptions for the storageclass
resource "kubernetes_storage_class_v1" "example" {
depends_on = [
helm_release.netapphelm,
kubernetes_secret_v1.netapp,
time_sleep.wait_600_seconds
]
metadata {
name = var.storageclass
}
storage_provisioner = "csi.trident.netapp.io"
reclaim_policy = "Delete"
volume_binding_mode = "WaitForFirstConsumer"
allow_volume_expansion = var.allow_volume_exspansion
parameters = {
backendType = "azure-netapp-files"
fsType: "nfs"
}
mount_options = ["vers=4.1"]also how can I open a netapp support ticket, it doesn't seem to let me open one on the site?
dc232
commented
2 years ago Hey @balaramesh ignore the above message I have managed to fix that one
I found in the logs for wordpress bitnami helm chart
Warning FailedMount 75s (x9 over 3m24s) kubelet MountVolume.SetUp failed for volume "pvc-f5b15aae-b758-4126-9012-c7f2642d6d36" : rpc error: code = Internal desc = error mounting NFS volume 192.1.0.4:/pvc-f5b15aae-b758-4126-9012-c7f2642d6d36 on mountpoint /var/lib/kubelet/pods/9627f587-b082-4b4b-87aa-ea81151d73f8/
volumes/kubernetes.io~csi/pvc-f5b15aae-b758-4126-9012-c7f2642d6d36/mount: exit status 32was wondering what things I could to try and debug I have VNET peering enabled have 111 and 2049 NFS enabled in the NSG I don't see any traffic flowing through and i don't have a firewall in the resource group is there anything else that I can check that springs to mind?
via prometheus bitnami helm chart
vents:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 14m default-scheduler Successfully assigned monitoring/prometheus-prometheus-kube-prometheus-prometheus-0 to aks-akscluster-24936088-vmss000000
Normal SuccessfulAttachVolume 14m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-bd2ecd02-c760-4285-9ee3-b2c5b8c17219"
Warning FailedMount 10m kubelet Unable to attach or mount volumes: unmounted volumes=[prometheus-prometheus-kube-prometheus-prometheus-db], unattached volumes=[prometheus-prometheus-kube-prometheus-prometheus-db web-config config config-out prometheus-prometheus-kube-prometheus-prometheus-rulefiles
-0 kube-api-access-np5kb tls-assets]: timed out waiting for the condition
Warning FailedMount 7m59s kubelet Unable to attach or mount volumes: unmounted volumes=[prometheus-prometheus-kube-prometheus-prometheus-db], unattached volumes=[config-out prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0 kube-api-access-np5kb tls-assets prometheus-prometheus-kube-prometh
eus-prometheus-db web-config config]: timed out waiting for the condition
Warning FailedMount 72s (x4 over 12m) kubelet Unable to attach or mount volumes: unmounted volumes=[prometheus-prometheus-kube-prometheus-prometheus-db], unattached volumes=[config config-out prometheus-prometheus-kube-prometheus-prometheus-rulefiles-0 kube-api-access-np5kb tls-assets prometheus-prometheus-kube-
prometheus-prometheus-db web-config]: timed out waiting for the condition
Warning FailedMount 12s (x15 over 14m) kubelet MountVolume.SetUp failed for volume "pvc-bd2ecd02-c760-4285-9ee3-b2c5b8c17219" : rpc error: code = Internal desc = error mounting NFS volume 192.1.0.4:/pvc-bd2ecd02-c760-4285-9ee3-b2c5b8c17219 on mountpoint /var/lib/kubelet/pods/d297951f-6f16-4e1b-8855-83293870846a/v
olumes/kubernetes.io~csi/pvc-bd2ecd02-c760-4285-9ee3-b2c5b8c17219/mount: exit status 32when logging into the node I found that the CSI does not seem to be creating the needed folders to mount the pod via the kubelet events
Apr 02 10:38:50 aks-akscluster-16723567-vmss000000 kubelet[3301]: E0402 10:38:50.003698 3301 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/csi/csi.trident.netapp.io^pvc-4d2b85eb-3de7-4524-825a-8b216e045418 podName: nodeName:}" failed. No
retries permitted until 2022-04-02 10:40:52.003671542 +0000 UTC m=+2038.016591882 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "pvc-4d2b85eb-3de7-4524-825a-8b216e045418" (UniqueName: "kubernetes.io/csi/csi.trident.netapp.io^pvc-4d2b85eb-3de7-
4524-825a-8b216e045418") pod "prometheus-prometheus-kube-prometheus-prometheus-0" (UID: "929e1d86-7415-43ab-a814-36f7d3577d25") : rpc error: code = Internal desc = error mounting NFS volume 192.1.0.4:/pvc-4d2b85eb-3de7-4524-825a-8b216e045418 on mountpoint /var/lib/kub
elet/pods/929e1d86-7415-43ab-a814-36f7d3577d25/volumes/kubernetes.io~csi/pvc-4d2b85eb-3de7-4524-825a-8b216e045418/mount: exit status 32root@aks-akscluster-16723567-vmss000000:~# ls -l /var/lib/kubelet/pods/929e1d86-7415-43ab-a814-36f7d3577d25/volumes/kubernetes.io~csi/
total 0tried
root@aks-akscluster-16723567-vmss000000:~# mkdir -p /var/lib/kubelet/pods/929e1d86-7415-43ab-a814-36f7d3577d25/volumes/kubernetes.io~csi/pvc-4d2b85eb-3de7-4524-825a-8b216e045418/mount
root@aks-akscluster-16723567-vmss000000:~# ls -l /var/lib/kubelet/pods/929e1d86-7415-43ab-a814-36f7d3577d25/volumes/kubernetes.io~csi/pvc-4d2b85eb-3de7-4524-825a-8b216e045418
total 4
drwxr-x--- 2 root root 4096 Apr 2 10:55 mount
root@aks-akscluster-16723567-vmss000000:~# chmod 0755 /var/lib/kubelet/pods/929e1d86-7415-43ab-a814-36f7d3577d25/volumes/kubernetes.io~csi/pvc-4d2b85eb-3de7-4524-825a-8b216e045418/mountafter a while it is just cleared?
root@aks-akscluster-16723567-vmss000000:/var/lib/kubelet/pods/929e1d86-7415-43ab-a814-36f7d3577d25/volumes/kubernetes.io~csi# ls -l pvc-4d2b85eb-3de7-4524-825a-8b216e045418/mount
total 0
root@aks-akscluster-16723567-vmss000000:/var/lib/kubelet/pods/929e1d86-7415-43ab-a814-36f7d3577d25/volumes/kubernetes.io~csi# ls -l pvc-4d2b85eb-3de7-4524-825a-8b216e045418/mount
ls: cannot access 'pvc-4d2b85eb-3de7-4524-825a-8b216e045418/mount': No such file or directorywhilst I understand this may be a network issue of some sort i still find it baffling that the required folders are not being created by the CSI
dc232
commented
2 years ago @balaramesh looking into the mount on the node the mount seems to read tempfs
root@aks-akscluster-16723567-vmss000000:~# mount | grep -i 929e1d86-7415-43ab-a814-36f7d3577d25
tmpfs on /var/lib/kubelet/pods/929e1d86-7415-43ab-a814-36f7d3577d25/volumes/kubernetes.io~secret/config type tmpfs (rw,relatime,size=5493280k)
tmpfs on /var/lib/kubelet/pods/929e1d86-7415-43ab-a814-36f7d3577d25/volumes/kubernetes.io~secret/config type tmpfs (rw,relatime,size=5493280k)
tmpfs on /var/lib/kubelet/pods/929e1d86-7415-43ab-a814-36f7d3577d25/volumes/kubernetes.io~projected/tls-assets type tmpfs (rw,relatime,size=5493280k)
tmpfs on /var/lib/kubelet/pods/929e1d86-7415-43ab-a814-36f7d3577d25/volumes/kubernetes.io~projected/tls-assets type tmpfs (rw,relatime,size=5493280k)
tmpfs on /var/lib/kubelet/pods/929e1d86-7415-43ab-a814-36f7d3577d25/volumes/kubernetes.io~secret/web-config type tmpfs (rw,relatime,size=5493280k)
tmpfs on /var/lib/kubelet/pods/929e1d86-7415-43ab-a814-36f7d3577d25/volumes/kubernetes.io~secret/web-config type tmpfs (rw,relatime,size=5493280k)
tmpfs on /var/lib/kubelet/pods/929e1d86-7415-43ab-a814-36f7d3577d25/volumes/kubernetes.io~projected/kube-api-access-xshcg type tmpfs (rw,relatime,size=5493280k)
tmpfs on /var/lib/kubelet/pods/929e1d86-7415-43ab-a814-36f7d3577d25/volumes/kubernetes.io~projected/kube-api-access-xshcg type tmpfs (rw,relatime,size=5493280k)is there meant to be a corresponding nfs mount to each pod or is it just a single mount? I don't see the netapp interface ip in the mount for the volume, i.e if I run mount against the IP which in this case is 192.1.0.4 i would expect to see something like this via the mount command on the node
192.1.0.4:/pvc-4d2b85eb-3de7-4524-825a-8b216e045418 on /root/pvc-4d2b85eb-3de7-4524-825a-8b216e045418 type nfs4 (rw,relatime,vers=4.1,rsize=65536,wsize=65536,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.1.0.4,local_lock=none,addr=192.1.0.4)if the above is typed in manually and the directory created i am able to download a file to the folder as illustrated below
root@aks-akscluster-16723567-vmss000000:~/pvc-4d2b85eb-3de7-4524-825a-8b216e045418# wget https://speed.hetzner.de/10GB.bin
--2022-04-02 11:16:55-- https://speed.hetzner.de/10GB.bin
Resolving speed.hetzner.de (speed.hetzner.de)... 88.198.248.254, 2a01:4f8:0:59ed::2
Connecting to speed.hetzner.de (speed.hetzner.de)|88.198.248.254|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 10485760000 (9.8G) [application/octet-stream]
Saving to: ‘10GB.bin’
10GB.bin 14%[=====================> ] 1.38G 1.20MB/s eta 2m 24s and I can then see the azure netapp file volume being utilised This leads me to believe here is a bug in the way the CSI is provisioning the volumes for consumption or the CRD which handles this is simply missing,
can I confirm that the helm chart at this repo location bundles the CRD's in with the install, I'm assuming they are installed automatically
]
name = "netapp-trident-operator"
namespace = kubernetes_namespace.netapp.metadata[0].name
repository = "https://netapp.github.io/trident-helm-chart"
chart = "trident-operator"
}also how des the CSI recognise the netapp interface IP and what does it need in order to successfully bind with it? and does it matter which namespace the netapp backend config is deployed into?
dc232
commented
2 years ago interestingly I have tried via this kubectl manifest and can form that this made the file called test
apiVersion: v1
kind: Pod
metadata:
name: test
labels:
app.kubernetes.io/name: alpine
app.kubernetes.io/part-of: kubernetes-complete-reference
app.kubernetes.io/created-by: ssbostan
spec:
containers:
- name: alpine
image: alpine:latest
command:
- touch
- /data/test
volumeMounts:
- name: nfs-volume
mountPath: /data
volumes:
- name: nfs-volume
nfs:
server: 192.1.0.4
path: /pvc-4d2b85eb-3de7-4524-825a-8b216e045418
readOnly: noroot@aks-akscluster-16723567-vmss000000:~/pvc-4d2b85eb-3de7-4524-825a-8b216e045418# ls -l
total 0
root@aks-akscluster-16723567-vmss000000:~/pvc-4d2b85eb-3de7-4524-825a-8b216e045418# ls -l
total 0
-rw-r--r-- 1 nobody 4294967294 0 Apr 2 13:33 test
root@aks-akscluster-16723567-vmss000000:~/pvc-4d2b85eb-3de7-4524-825a-8b216e045418# I then revised the configuration to
apiVersion: v1
kind: Pod
metadata:
name: test
labels:
app.kubernetes.io/name: alpine
app.kubernetes.io/part-of: kubernetes-complete-reference
app.kubernetes.io/created-by: ssbostan
spec:
containers:
- name: alpine
image: alpine:latest
command:
- cat
- /data/test
volumeMounts:
- name: nfs-volume
mountPath: /data
volumes:
- name: nfs-volume
nfs:
server: 192.1.0.4
path: /pvc-4d2b85eb-3de7-4524-825a-8b216e045418
readOnly: nowhilst modifying the test file to contain
root@aks-akscluster-16723567-vmss000000:~/pvc-4d2b85eb-3de7-4524-825a-8b216e045418# echo "this is a test to see if the pod is able to read from the netapp mountpoint" > test
root@aks-akscluster-16723567-vmss000000:~/pvc-4d2b85eb-3de7-4524-825a-8b216e045418# cat test
this is a test to see if the pod is able to read from the netapp mountpointwhen reading back the file from the pod it displayed the correct message when bypassing storageclass as the volume had been provisioned by the netapp backend previously (anj)
PS C:\Users\danny> kubectl logs test --kubeconfig config
this is a test to see if the pod is able to read from the netapp mountpointThis suggests a a problem with the storage class when communicating with the Netapp backend @balaramesh is there any additional documentation on how to configure the storage class to work alongside the Netapp backend?
clintonk
commented
1 year ago It appears ANF created volumes with no mount targets, which will cause the exit status 32 errors. Is this still an issue? If this happens again, please open an Azure support ticket.
dc232
commented
1 year ago I don't the issue exists anymore, it did work when I have tested it in the past
clintonk
commented
1 year ago Thanks, @dc232. Closing for now.
Describe the bug it appears that the CSI is unable to mount the NFS volume to a PVC in azure, I am unsure as to why this is as NFS is installed by default in every node in the AKS vmss cluster
Environment Provide accurate information about the environment to help us reproduce the issue.
To Reproduce Deploy WordPress bitnami helm chart as RWX with cluster autoscaling enabled along with Promeuthus and grafana This results in error 32 codes it also looks like Trident expects a minimum of 100GB to be provisioned per volume but I think this value should really be 100G or 100Gi, I'm not sure exactly how to provision 100GB of storage via kubernetes
Expected behavior NFS volume in Netapp mounted to the pod in running state
Additional context
The wordpress-memcached-774f78cc49-48vsx is bizarre that it's running as this is using the same storage class in the WordPress bitnami helmchart
additional terraform provisioning
Was wondering does the CSI helm chat use the ports 31579 and 32165 in any way and can these ports be changed to something else via the helm chart?