search

kubernetes-sigs / aws-efs-csi-driver

CSI Driver for Amazon EFS https://aws.amazon.com/efs/
Apache License 2.0
723 stars 554 forks source link

Not able to mount volume: Please set the "region" parameter in the efs-utils configuration file #742

Closed TBeijen closed 1 year ago

TBeijen commented 2 years ago

/kind bug

What happened?

A PVC provisioned by aws-efs-csi-driver, previously mounting fine, fails to mount after pod restart due to an error seemingly originating from efs-utils:

Error retrieving region. Please set the "region" parameter in the efs-utils configuration file.

What you expected to happen?

PVC mounting fine

How to reproduce it (as minimally and precisely as possible)?

Not (yet) clear since all other PVCs using same storageclass, on same and identical clusters so far run fine. Error seems specific for the volume since on replacing pod it persists (PVC used by single statefulset pod).

Anything else we need to know?:

Environment

Recently upgraded efs-csi-driver from 1.3.5 to 1.4.0. Affected PVC has been created by 1.4.0 but so have others that work fine.

Errors

# describe pod that needs the volume
 Normal   Scheduled    93s   default-scheduler  Successfully assigned sentry/sentry-kafka-2 to ip-100-64-61-170.eu-west-1.compute.internal
  Warning  FailedMount  93s   kubelet            MountVolume.SetUp failed for volume "pvc-d0c863ef-419a-4975-9b2d-a4f3b39b3f66" : rpc error: code = Internal desc = Could not mount "fs-0bb74173d3eec42e2:/" at "/var/lib/kubelet/pods/f81eaa1a-36cf-4ee4-ad3d-8b89ef40081b/volumes/kubernetes.io~csi/pvc-d0c863ef-419a-4975-9b2d-a4f3b39b3f66/mount": mount failed: exit status 1
Mounting command: mount
Mounting arguments: -t efs -o accesspoint=fsap-04ef996767a44a910,tls fs-0bb74173d3eec42e2:/ /var/lib/kubelet/pods/f81eaa1a-36cf-4ee4-ad3d-8b89ef40081b/volumes/kubernetes.io~csi/pvc-d0c863ef-419a-4975-9b2d-a4f3b39b3f66/mount
Output: Error retrieving region. Please set the "region" parameter in the efs-utils configuration file.

# efs-csi-node
efs-csi-node-577kt efs-plugin Mounting arguments: -t efs -o accesspoint=fsap-04ef996767a44a910,tls fs-0bb74173d3eec42e2:/ /var/lib/kubelet/pods/f81eaa1a-36cf-4ee4-ad3d-8b89ef40081b/volumes/kubernetes.io~csi/pvc-d0c863ef-419a-4975-9b2d-a4f3b39b3f66/mount
efs-csi-node-577kt efs-plugin Output: Error retrieving region. Please set the "region" parameter in the efs-utils configuration file.
efs-csi-node-577kt efs-plugin W0726 04:40:24.348078       1 node.go:158] Use of 'tls' under mountOptions is deprecated with this driver since tls is enabled by default. To disable it, set encrypt in transit in the volumeContext, e.g. 'encryptInTransit: true'

Storageclass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    meta.helm.sh/release-name: aws-efs-csi-driver
    meta.helm.sh/release-namespace: kube-system
    storageclass.kubernetes.io/is-default-class: "true"
  creationTimestamp: "2022-04-13T14:48:13Z"
  labels:
    app.kubernetes.io/managed-by: Helm
  name: efs
  resourceVersion: "77747"
  uid: 264ae9fd-6f04-4e86-b2f0-092b38c16ebb
mountOptions:
- tls
parameters:
  basePath: /dynamic_provisioning
  directoryPerms: "700"
  fileSystemId: fs-0bb74173d3eec42e2
  gidRangeEnd: "50000"
  gidRangeStart: "40000"
  provisioningMode: efs-ap
provisioner: efs.csi.aws.com
reclaimPolicy: Delete
volumeBindingMode: Immediate
TBeijen commented 2 years ago

Deleting the specific efs-csi-node pod, then after it got replaced killing the statefulset pod that tried to mount the PVC, seems to have 'fixed' things.

Not knowing the ins and outs of hwo the efs-csi-node interact, it kinda looks like the pod got into a bad state, losing it's ability (or stored state) that let's it determine the region.

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

njuettner commented 2 years ago

/remove-lifecycle stale

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented 1 year ago

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes-sigs/aws-efs-csi-driver/issues/742#issuecomment-1492968055): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.