kubernetes-sigs / aws-ebs-csi-driver

CSI driver for Amazon EBS https://aws.amazon.com/ebs/
Apache License 2.0
976 stars 787 forks source link

Intermittent Volume Mounting Error #1992

Open atilsensalduz opened 6 months ago

atilsensalduz commented 6 months ago

Hello everyone, I've been encountering an intermittent issue in my Kubernetes environment especially in under load and I was hoping to get some insight into what might be causing it. Occasionally, I see the following error message in the Kubernetes events:

Unable to attach or mount volumes: unmounted volumes=[config-volume-5favqh58-jv5hu1f5 keys-volume-5favqh58-jv5hu1f5 persistence-jv5hu1f5 kube-api-access-bf5b7], 
unattached volumes=[], failed to process volumes=[persistence-jv5hu1f5]: 
error processing PVC -5favqh58/persistence-jv5hu1f5-5favqh58-jv5hu1f5-46-1: failed to fetch PVC from API server: 
persistentvolumeclaims \"persistence-jv5hu1f5-5favqh58-jv5hu1f5-46-1\" is forbidden: User \"system:node:ip-10-10-10-10.us-east-2.compute.internal\" 
cannot get resource \"persistentvolumeclaims\" in API group \"\" in the namespace \"5favqh58\": 
no relationship found between node 'ip-10-10-10-10.us-east-2.compute.internal' and this object

It doesn't happen all the time, which makes it more perplexing. I've been suspecting that it might be related to some rate limits or perhaps an issue with handling requests from the EKS control plane side, but I'm not entirely sure. Has anyone else encountered a similar issue before? Or does anyone have any insights into what might be causing this error? Any help or suggestions would be greatly appreciated! Thanks in advance

/triage support

k8s-ci-robot commented 6 months ago

@atilsensalduz: The label(s) triage/support cannot be applied, because the repository doesn't have them.

In response to [this](https://github.com/kubernetes-sigs/aws-ebs-csi-driver/issues/1992): >Hello everyone, I've been encountering an intermittent issue in my Kubernetes environment especially in under load and I was hoping to get some insight into what might be causing it. Occasionally, I see the following error message in the Kubernetes events: > >``` >Unable to attach or mount volumes: unmounted volumes=[config-volume-5favqh58-jv5hu1f5 keys-volume-5favqh58-jv5hu1f5 persistence-jv5hu1f5 kube-api-access-bf5b7], >unattached volumes=[], failed to process volumes=[persistence-jv5hu1f5]: >error processing PVC -5favqh58/persistence-jv5hu1f5-5favqh58-jv5hu1f5-46-1: failed to fetch PVC from API server: >persistentvolumeclaims \"persistence-jv5hu1f5-5favqh58-jv5hu1f5-46-1\" is forbidden: User \"system:node:ip-10-10-10-10.us-east-2.compute.internal\" >cannot get resource \"persistentvolumeclaims\" in API group \"\" in the namespace \"5favqh58\": >no relationship found between node 'ip-10-10-10-10.us-east-2.compute.internal' and this object >``` > >It doesn't happen all the time, which makes it more perplexing. I've been suspecting that it might be related to some rate limits or perhaps an issue with handling requests from the EKS control plane side, but I'm not entirely sure. Has anyone else encountered a similar issue before? Or does anyone have any insights into what might be causing this error? Any help or suggestions would be greatly appreciated! Thanks in advance > >/triage support > Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
k8s-triage-robot commented 3 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 2 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

AndrewSirenko commented 2 months ago

/remove-lifecycle rotten