Closed diamonwiggins closed 8 months ago
Hi @diamonwiggins, ref:issues-2629 Did it 'FailedMount' happen for a long time before you created the volumeattachment object?
@mantissahz I can get clarification from the end user if the amount of time is relevant. At the very least, 5 minutes had passed, but it's likely that much more time had passed before the user was assisted with manually creating the volume attachment.
Also worth noting, this customer is on 1.1.2 where #2629 is supposedly fixed. Happy to provide any other information that could help track this down.
It could take up to 6 or 7 minutes for Kubernetes to retry creating the volumeattachment object.
How long did the node go down?
@PhanLe1010 The node went down for only minutes. Maybe 5 minutes or so. However it was a full 24 hours before the user manually created the VolumeAttachment objects.
Is there any additional information I can gather to assist here?
@diamonwiggins I don't seem to figure out why the VA is deleted and never been created automatically as you mentioned earlier that the VA removal feature was removed since Longhorn 1.1.2.
I would suggest upgrading to a newer stable Longhorn version (1.1.3 or 1.2.5) and report back when you hit the issue again
@PhanLe1010 Understood. We've seen a similar issue with another customer after a reboot. The error is slightly different this time with:
MountVolume.WaitForAttach failed for volume "pvc-e25ec426-043d-496d-9ddd-e4920e8c1096" : volume pvc-e25ec426-043d-496d-9ddd-e4920e8c1096 has GET error for volume attachment csi-845807a0d4e3617baaadf26f975d24db606458cb640455aaac527298e9a2c4bd: volumeattachments.storage.k8s.io "csi-845807a0d4e3617baaadf26f975d24db606458cb640455aaac527298e9a2c4bd" is forbidden: User "system:node:ip-10-0-1-200" cannot get resource "volumeattachments" in API group "storage.k8s.io" at the cluster scope: no relationship found between node 'ip-10-0-1-200' and this object
We've confirmed that node names and IP address had not changed, and our customer was able to reproduce this on two separate environments.
longhorn-support-bundle_b27f4748-ec4a-45a1-8d75-04fe278d3584_2022-09-07T18-57-23Z (1).zip
If this warrants a separate github issue let me know and i'll get one opened up.
cc @joshimoo
we see this issue so often with longhorn and today also we had a repro where on node restart in a multi-node environment alertmanager statefulset related pods were not able to mount PVCs even after 30-40 minutes and we see this issue with both deployments and statefulsets
This is happening with longhorn 1.3.1 also and this repro is with longhorn 1.3.1 itself
When ever this happens we scaledown the workload replicas to 0 and scale it back such that volume attachment flow gets triggered again but this is not an acceptable solution while running production workloads
{
"apiVersion": "v1",
"count": 60,
"eventTime": null,
"firstTimestamp": "2022-10-12T06:05:07Z",
"involvedObject": {
"apiVersion": "v1",
"kind": "Pod",
"name": "alertmanager-rancher-monitoring-alertmanager-1",
"namespace": "cattle-monitoring-system",
"resourceVersion": "83537",
"uid": "2088fca6-b6cb-458f-8297-44fa477b0e81"
},
"kind": "Event",
"lastTimestamp": "2022-10-12T07:50:59Z",
"message": "MountVolume.WaitForAttach failed for volume \"pvc-84933541-a66d-4ca2-a710-6db17e6643ba\" : volume pvc-84933541-a66d-4ca2-a710-6db17e6643ba has GET error for volume attachment csi-0c400de43ff27c65fa12afab1248675317dbb2b8fc07ae6582df5ce218fa6ff7: volumeattachments.storage.k8s.io \"csi-0c400de43ff27c65fa12afab1248675317dbb2b8fc07ae6582df5ce218fa6ff7\" is forbidden: User \"system:node:server1\" cannot get resource \"volumeattachments\" in API group \"storage.k8s.io\" at the cluster scope: no relationship found between node 'server1' and this object",
"metadata": {
"creationTimestamp": "2022-10-12T06:05:07Z",
"name": "alertmanager-rancher-monitoring-alertmanager-1.171d3d2e89354c2e",
"namespace": "cattle-monitoring-system",
"resourceVersion": "167371",
"uid": "1874d80b-43e9-4242-9df3-bc39b68c0cc1"
},
"reason": "FailedMount",
"reportingComponent": "",
"reportingInstance": "",
"source": {
"component": "kubelet",
"host": "server1"
},
"type": "Warning"
},
@rajivml
Could you help us troubleshoot by providing the reproducing steps and env information (or provide us an env)?
HI @PhanLe1010
We are seeing it on both single node and multi-node environments
I will share an environment for your offline analysis via DM over slack
Longhorn version: 1.3.1 Installation method (e.g. Rancher Catalog App/Helm/Kubectl): Helm Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: RKE2 Number of management node in the cluster: 3 nodes which acts as both master + worker Number of worker node in the cluster: 3 nodes which acts as both master + worker Node config: 32 Core 128GB RAM OS type and version: RHEL CPU per node: 32 Memory per node: 128GB RAM Disk type(e.g. SSD/NVMe): SSD Network bandwidth between the nodes: Azure Provided Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): Azure Disks Number of Longhorn volumes in the cluster: Around 20
I saw this today also. It was after I upgraded k3s from 1.23.4 -> 1.23.13, but that may be coincidence.
Specifically, I saw the User "USER" cannot get resource "volumeattachments" in API group "storage.k8s.io" at the cluster scope
error
User "USER" cannot get resource "volumeattachments" in API group "storage.k8s.io" at the cluster scope
This error is not related to this issue. It indicate that the client is missing the RBAC permission. Where did you see that error (from which pods)?
I have the same issue. I have a two node RPI cluster. Any time master reboots, I get something like:
Warning FailedMount 37s kubelet MountVolume.WaitForAttach failed for volume "ghost-db" : volume ghost-db has GET error for volume attachment csi-be2cb4dfc03d99eef9aa0e05cb28e59ac52f0c0c5e832c68d142a2ba76827bdb: volumeattachments.storage.k8s.io "csi-be2cb4dfc03d99eef9aa0e05cb28e59ac52f0c0c5e832c68d142a2ba76827bdb" is forbidden: User "system:node:pi4" cannot get resource "volumeattachments" in API group "storage.k8s.io" at the cluster scope: no relationship found between node 'pi4' and this object
Hi I have same issue
MountVolume.WaitForAttach failed for volume "pvc-xx" : volume vol-xx has GET error for volume attachment csi-xx: volumeattachments.storage.k8s.io "csi-xx" is forbidden: User "system:node:ip-xx.compute.internal" cannot get resource "volumeattachments" in API group "storage.k8s.io" at the cluster scope: no relationship found between node 'ip-xx.compute.internal' and this object
But we scaledown the statefulset replicas to 0 and scale it back such that volume attachment flow gets triggered again
Also run into this issue. Can confirm that workaround suggested by @simonreddy2001 works.
We need to have a resilience way to recover from this automatically.
cc @derekbit @shuo-wu @PhanLe1010
@diamonwiggins @hedefalk @simonreddy2001 @Orhideous @rajivml I tried to reproduce the issue using Longhorn v1.3.2 and a StatefulSet with 2 replicas on a 2-node cluster. Reboot the two nodes repeatedly, but still cannot reproduce the issue.
Could you please provide the reproducing steps? If you run into the issue again, could you provide a support bundle as well? Thanks.
Theoretically, this issue could be very well related to the upstream issue https://github.com/kubernetes/kubernetes/issues/120571.
However, attempting to reproduce using similar steps as in the upstream issue yields no success. The attempted reproducing steps are:
phanle1010/longhorn-manager:v1.5.3-injected-detach-error
. Thi longhorn-manager image added the logic to artificially inject the detach error into longhorn-csi-plugin to simulate a temporary detach error. This is the code https://github.com/PhanLe1010/longhorn-manager/commit/a29962e468806eb8209ad45b53ec4be204d4266dapiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: '1'
generation: 7
labels:
workload.user.cattle.io/workloadselector: apps.deployment-default-test-dep
name: test-dep
namespace: default
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
workload.user.cattle.io/workloadselector: apps.deployment-default-test-dep
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
workload.user.cattle.io/workloadselector: apps.deployment-default-test-dep
namespace: default
spec:
affinity: {}
containers:
- image: ubuntu:xenial
imagePullPolicy: Always
name: container-0
resources: {}
securityContext:
allowPrivilegeEscalation: false
privileged: false
readOnlyRootFilesystem: false
runAsNonRoot: false
stdin: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
tty: true
volumeMounts:
- mountPath: /mnt
name: vol-7rasu
dnsPolicy: ClusterFirst
nodeName: phan-v603-pool2-e46dd713-75pnq
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
volumes:
- name: vol-7rasu
persistentVolumeClaim:
claimName: test-pvc
Use this script to simulate the reproducing steps in the upstream issue
#!/bin/bash
set -o errexit
set -o nounset
set -o pipefail
set -x
# Inject detach error
kubectl -n longhorn-system patch -p '{"value": "26"}' --type=merge lhs storage-minimal-available-percentage
# Run application pod
kubectl scale deployment test-dep --replicas 1
kubectl wait --for=condition=available deployment/test-dep
# Delete the app
kubectl scale deployment test-dep --replicas 0
kubectl wait --for=delete pod --selector=workload.user.cattle.io/workloadselector=apps.deployment-default-test-dep
# Wait for detach error
while true; do
if kubectl get volumeattachment -o json | grep "Simulated detach error"; then
break
fi
echo "Waiting for volumeAttachment to get error..."
sleep 1
done
# Kill KCM
kubectl -n kube-system delete pod -l component=kube-controller-manager
sleep 2
# Start a new KCM
kubectl -n kube-system wait --for condition=Ready=true pod --selector=component=kube-controller-manager
# there is no way how to wait for KCM to process the volumeattachment...
sleep 13
# Create a new pod *after* KCM started processing volumeattachments
kubectl scale deployment test-dep --replicas 1
sleep 1
kubectl wait --for condition=PodScheduled=true pod --selector=workload.user.cattle.io/workloadselector=apps.deployment-default-test-dep
# Stop injecting errors to detach
kubectl -n longhorn-system patch -p '{"value": "25"}' --type=merge lhs storage-minimal-available-percentage
# Now, the second pod should start, but it's stuck at "no relationship found between node '127.0.0.1' and this object"
Even though we are not able to reproduce the upstream issue, from code analysis, I do think that the race condition in the upstream issue COULD be the root cause of this ticket. The upstream issue is fixed in:
Therefore, I think the next step for this ticket would be:
WDYT @derekbit @innobead @ejweber ?
Additionally, from code analysis, I think the workaround may be to scale down the workload, wait for the workload to be fully terminated, then scale back the workload again. Kube-controller-manager should be able to recreate the VolumeAttachment for the new pod
Hi, I have face the similar issue in Kubernetes v1.29
I'm seeing the same behaviour when using Mayastor. In my case I drained the node of Mayastor volumes, restarted the Mayastor pod (openebs-io-engine-xxx
), the uncordoned the node to Mayastor volumes. I then noted that some of the Stateful set volumes were stuck in 'ContainerCreating', reporting an io error
with no further details.
The OpenEBS CSI controller was reporting:
I0511 17:34:46.274399 1 csi_handler.go:234] Error processing "csi31ad7af564f89fe04d71d5cc0e2240ee1f5b73d9da88f3e933c1b26d9f501219": failed to detach: could not mark as detached: volumeattachments.storage.k8s.io"csi31ad7af564f89fe04d71d5cc0e2240ee1f5b73d9da88f3e933c1b26d9f501219" not found
Scaling down the StatefulSet to 0, then scaling back up resolved the issue.
I know Mayastore is an entirely different project, but I thought this would be helpful information for 1) anyone also googling their way here, and 2) adding information towards the "is this an upstream issue" question?
Describe the bug
After a reboot of a node in a 4 node cluster a user is seeing the following:
To recover, the user had to create the volumeattachment object manually for the Pod to mount its storage again
To Reproduce
I have not been able to reproduce this yet unfortunately
Expected behavior
A pod can successfully mount its storage despite a node reboot in the cluster
Log or Support bundle
longhorn-support-bundle_a8118729-480f-4d38-9b91-26a755d2e0cc_2022-06-28T20-34-47Z.zip
Environment