`"validations.kong.konghq.com" denied the request: consumer already exists` error in isolated namespaces

[metheu]

Is there an existing issue for this?

• [X] I have searched the existing issues

Kong version ($ kong version)

Kong 3.1

Current Behavior

In our environment we have deployed the same consumers in different namespaces. This has previously worked before but recently we were doing changes around our Kong consumers and on adding new environments we are getting the following error: one or more objects failed to apply, reason: admission webhook "validations.kong.konghq.com" denied the request: consumer already exists

Kong logs that are probably related:

time="2024-09-17T09:30:18Z" level=error msg="failed to run validation" component=admission-server error="unique key constraint violated for key"

We are aware of the documentation that mentions username should be unique but this method of isolation has worked before and should probably continue to do so. Our environments should be a easily reproducible as possible.

Expected Behavior

No response

Steps To Reproduce

No response

Anything else?

Cluster information: EKS v1.28

[randmonkey]

@metheu could you please provide the versions of Kong and Kong ingress controller before/after the issue happens? Also, different consumers in different Kong workspaces not different k8s namespaces can share same username, and k8s object name and username in consumer (from username of KongConsumer object) are different things.

[metheu]

@metheu could you please provide the versions of Kong and Kong ingress controller before/after the issue happens? Also, different consumers in different Kong workspaces not different k8s namespaces can share same username, and k8s object name and username in consumer (from username of KongConsumer object) are different things.

hey @randmonkey, thanks for reply. We did not to any upgrades to Kong ingress controller through this period. We are still running the same version of kong (app.kubernetes.io/version=3.1).

We do not utilize Kong workspaces - we are using the kind object KongConsumer. This is an example of how we use it.

apiVersion: configuration.konghq.com/v1
kind: KongConsumer
metadata:
  name: api-consumer-apikey
  annotations:
    kubernetes.io/ingress.class: kong
username: api-consumer-apikey
credentials:
  - api-consumer-apikey

This object is replicated across our environments in order to stick to the same global schema. This is how it was designed and recently, we discovered that whenever we redeployed a new environment(i.e. a new kube namespace) we are getting the above error. Our only assumption is that we have recently updated our version of EKS and that might have moved something but we can't point it out.

[randmonkey]

@metheu Another possibility is that you deployed different KIC and Kong gateway instances in different namespaces and KIC only controls the KongConsumer in its namespace and send them to managed Kong gateways. In this scenario, different KongConsumers are translated to Kong configuration for different Kong instances and they will not affect each other. To confirm that, could you please tell us the kubernetes.io/ingress.class annotation of affected KongConsumers? And also please provide version of KIC and Kong gateway, by container images.

[metheu]

@randmonkey can you please elaborate on the difference between these 2 components? We have deployed in a separate namespace(named kong) 2 pods with 3 containers - ingress-controller, proxy and clear-stale-pid(init-container). Container image is kong/kubernetes-ingress-controller:2.8and for proxy kong:3.1. We do not have any other kong resource except the above.

The annotation on the KongConsumers is kubernetes.io/ingress.class":"kong". I checked all KongConsumers on the cluster and they all hold the same class(.i.e kong).

[github-actions[bot]]

This issue is marked as stale because it has been open for 14 days with no activity.

[metheu]

@randmonkey hey, did you have time to look into this? What can we do to try and diagnose the issue?

[randmonkey]

@metheu If the KongConsumers all have the same annotation kubernetes.io/ingress.class, they should have unique username each.