search

kubernetes-sigs / aws-ebs-csi-driver

CSI driver for Amazon EBS https://aws.amazon.com/ebs/
Apache License 2.0
973 stars 786 forks source link

Parameters on this idempotent request are inconsistent with parameters used in previous request #1916

Closed ananthbh-te closed 2 months ago

ananthbh-te commented 7 months ago

/kind bug

**When i try to create a new volume with EBS csi driver installed, i'm facing this issue.

This was not the case before installing EBS CSI driver. **

EBS csi driver should create the volume

PVC events

Events:
  Type     Reason              Age   From                                                                                      Message
  ----     ------              ----  ----                                                                                      -------
  Warning  ProvisioningFailed  68s   ebs.csi.aws.com_ebs-csi-controller-847d68f976-tjn7h_a6ba8172-6d47-4c2d-a5a0-e52b7bd611af  failed to provision volume with StorageClass "influxdb-data-kube-services-pvc-class": rpc error: code = Internal desc = Could not create volume "pvc-d34ce440-7161-4a88-8363-001a057eca81": failed to get an available volume in EC2: InvalidVolume.NotFound: The volume 'vol-086af10dba8b4b9be' does not exist.
           status code: 400, request id: bf656776-28fa-469b-9ae1-248725b0a006
  Normal   Provisioning          4s (x7 over 72s)  ebs.csi.aws.com_ebs-csi-controller-847d68f976-tjn7h_a6ba8172-6d47-4c2d-a5a0-e52b7bd611af  External provisioner is provisioning volume for claim "kube-services/influxdb-data"
  Warning  ProvisioningFailed    4s (x6 over 67s)  ebs.csi.aws.com_ebs-csi-controller-847d68f976-tjn7h_a6ba8172-6d47-4c2d-a5a0-e52b7bd611af  failed to provision volume with StorageClass "influxdb-data-kube-services-pvc-class": rpc error: code = AlreadyExists desc = Could not create volume "pvc-d34ce440-7161-4a88-8363-001a057eca81": Parameters on this idempotent request are inconsistent with parameters used in previous request(s)
  Normal   ExternalProvisioning  3s (x7 over 72s)  persistentvolume-controller                                                               waiting for a volume to be created, either by external provisioner "ebs.csi.aws.com" or manually created by system administrator

logs of ebs-csi-controller pod

kubectl logs ebs-csi-controller-847d68f976-swkpq -n kube-system
Defaulted container "ebs-plugin" out of: ebs-plugin, csi-provisioner, csi-attacher, csi-snapshotter, csi-resizer, liveness-probe
I1108 18:21:18.713260       1 driver.go:75] "Driver Information" Driver="ebs.csi.aws.com" Version="v1.21.0"
ANANTHBH-M-9N44:cloud-components ananthbh$ kubectl logs ebs-csi-controller-847d68f976-tjn7h -n kube-system
Defaulted container "ebs-plugin" out of: ebs-plugin, csi-provisioner, csi-attacher, csi-snapshotter, csi-resizer, liveness-probe
I0201 06:29:22.915609       1 driver.go:75] "Driver Information" Driver="ebs.csi.aws.com" Version="v1.21.0"
I0201 06:29:46.561742       1 controller.go:146] "\"fstype\" is deprecated, please use \"csi.storage.k8s.io/fstype\" instead"
E0201 06:29:46.804122       1 driver.go:122] "GRPC error" err="rpc error: code = AlreadyExists desc = Could not create volume \"pvc-89862b9e-f96a-40c0-9176-9986c7d37b7a\": Parameters on this idempotent request are inconsistent with parameters used in previous request(s)"
I0201 06:29:47.806216       1 controller.go:146] "\"fstype\" is deprecated, please use \"csi.storage.k8s.io/fstype\" instead"
E0201 06:29:48.003343       1 driver.go:122] "GRPC error" err="rpc error: code = AlreadyExists desc = Could not create volume \"pvc-89862b9e-f96a-40c0-9176-9986c7d37b7a\": Parameters on this idempotent request are inconsistent with parameters used in previous request(s)"
I0201 06:29:50.004655       1 controller.go:146] "\"fstype\" is deprecated, please use \"csi.storage.k8s.io/fstype\" instead"
E0201 06:29:50.203728       1 driver.go:122] "GRPC error" err="rpc error: code = AlreadyExists desc = Could not create volume \"pvc-89862b9e-f96a-40c0-9176-9986c7d37b7a\": Parameters on this idempotent request are inconsistent with parameters used in previous request(s)"
I0201 06:29:54.205298       1 controller.go:146] "\"fstype\" is deprecated, please use \"csi.storage.k8s.io/fstype\" instead"
E0201 06:29:54.403250       1 driver.go:122] "GRPC error" err="rpc error: code = AlreadyExists desc = Could not create volume \"pvc-89862b9e-f96a-40c0-9176-9986c7d37b7a\": Parameters on this idempotent request are inconsistent with parameters used in previous request(s)"

Environment

ananthbh-te commented 7 months ago

I verified the EBS CSI driver policy and it seems fine - https://github.com/kubernetes-sigs/aws-ebs-csi-driver/issues/1140#issuecomment-1094892327

ConnorJC3 commented 7 months ago

Hi @ananthbh-te , what you're seeing is likely an asynchronous creation failure due to KMS encryption of a volume failing.

If you've confirmed your policy is correct, I would also confirm that the ARN you are passing in your storage class points to a valid and existing KMS key in the same region as your volume.

If that doesn't help, could you provide the StorageClass parameters that you are using to create the volume?

rifelpet commented 7 months ago

FWIW Kops' prow k/k e2e testing has seen this same flaky error (see the linked issue above)

https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/kops/16341/pull-kops-e2e-k8s-aws-calico/1756400222856024064

Feb 10 20:13:18.720: INFO: At 2024-02-10 20:08:18 +0000 UTC - event for aws46qr4: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
Feb 10 20:13:18.720: INFO: At 2024-02-10 20:08:18 +0000 UTC - event for aws46qr4: {ebs.csi.aws.com_ebs-csi-controller-84d99d85bf-gds8t_2697faee-bf5c-4097-9f9f-d34526ec0feb } Provisioning: External provisioner is provisioning volume for claim "volume-expand-6492/aws46qr4"
Feb 10 20:13:18.720: INFO: At 2024-02-10 20:08:18 +0000 UTC - event for aws46qr4: {persistentvolume-controller } ExternalProvisioning: Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
Feb 10 20:13:18.720: INFO: At 2024-02-10 20:08:22 +0000 UTC - event for aws46qr4: {ebs.csi.aws.com_ebs-csi-controller-84d99d85bf-gds8t_2697faee-bf5c-4097-9f9f-d34526ec0feb } ProvisioningFailed: failed to provision volume with StorageClass "volume-expand-6492cxrrm": rpc error: code = Internal desc = Could not create volume "pvc-8d24e563-c64c-45df-9f31-e7c3c9b8a994": failed to get an available volume in EC2: InvalidVolume.NotFound: The volume 'vol-029d5b5c03cd79b35' does not exist.
    status code: 400, request id: 15206e5b-b1fd-4254-8dea-1f8be92d4cb8
Feb 10 20:13:18.720: INFO: At 2024-02-10 20:08:24 +0000 UTC - event for aws46qr4: {ebs.csi.aws.com_ebs-csi-controller-84d99d85bf-gds8t_2697faee-bf5c-4097-9f9f-d34526ec0feb } ProvisioningFailed: failed to provision volume with StorageClass "volume-expand-6492cxrrm": rpc error: code = AlreadyExists desc = Could not create volume "pvc-8d24e563-c64c-45df-9f31-e7c3c9b8a994": Parameters on this idempotent request are inconsistent with parameters used in previous request(s)
k8s-triage-robot commented 4 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 3 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 2 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented 2 months ago

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes-sigs/aws-ebs-csi-driver/issues/1916#issuecomment-2218761028): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.