search

apache / apisix-ingress-controller

APISIX Ingress Controller for Kubernetes
https://apisix.apache.org/
Apache License 2.0
1.02k stars 343 forks source link

bug: cannot delete consumer if secret used in KeyAuth secret ref has been deleted first #1627

Closed ghost closed 1 year ago

ghost commented 1 year ago

Issue description

TL;DR

When deleting Consumer manifests - and - if using an authentication scheme such as KeyAuth which references a Kubernetes secret - if the secret has already been deleted within the environment, the consumer cannot be removed [and retries infinitely]

More Context:

We use argocd and helm charts to deploy our services. We manage ingress access in these charts by creating secrets and consumers. The secrets created are the API tokens that consumers refer to for KeyAuth. The secrets are managed by external-secrets and thus pre-exist in a secret manager. Deploying consumers is no issue - however, when we delete everything in a particular release - not all consumers are deleted [non-deterministic] and those that cannot be deleted enter into an infinite loop of retries.

They are still contained with etcd and therefore available on the APISIX api.

From a quick glance at the code for 1.5 - I believe this is happening due to the fact that in the environment, argocd can delete secrets before the ingress-controller has processed the consumer deletion event [race condition]

The ingress-controller tries to translate the consumer to be removed and fails to parse / translate due to the secret being missing.

The only way to remove the consumer entry is to manually delete it or restore the secret.

Useful links to functions

https://github.com/apache/apisix-ingress-controller/blob/32be235aaf1f571a9f70f6375459429cd4ecc1af/pkg/kube/translation/apisix_consumer.go#L68

https://github.com/apache/apisix-ingress-controller/blob/32be235aaf1f571a9f70f6375459429cd4ecc1af/pkg/kube/translation/plugin.go#L125

Environment

Minimal test code / Steps to reproduce

  1. Create Kubernetes Secret with property key=token
  2. Create Consumer with keyAuth reference to Kubernetes Secret
  3. Ensure syncrhonised with APISIX [check admin API or etcd]
  4. Delete Secret
  5. Delete Consumer

Actual result

Consumer is not deleted

Error log

error ingress/apisix_consumer.go:166 failed to translate ApisixConsumer {"error": "invalid key auth config: secret \"dev-api-key-gn78j-15\" not found"... [Apisix Consumer Object JSON is printed to screen - removed for brevity]

ingress/status.go:220 failed to record status change for ApisixConsumer {"error": "Operation cannot be fulfilled on apisixconsumers.apisix.apache.org \"t-gn78j-15\": StorageError: invalid object, Code: 4, Key: /registry/apisix.apache.org/apisixconsumers/t-gn78j/t-gn78j-15, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: e8fb79f7-e8e2-4e7c-9746-b17c8670f683, UID in object meta: ", "name": "t-gn78j-15", "namespace": "t-gn78j"}

Expected result

Consumer is deleted

tao12345666333 commented 1 year ago

thanks for the report, add this to my queue.

ghost commented 1 year ago

Thanks, @tao12345666333 - very happy to help test fixes.

github-actions[bot] commented 1 year ago

This issue has been marked as stale due to 90 days of inactivity. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please simply write any comment. Even if closed, you can still revive the issue at any time or discuss it on the dev@apisix.apache.org list. Thank you for your contributions.

github-actions[bot] commented 1 year ago

This issue has been closed due to lack of activity. If you think that is incorrect, or the issue requires additional review, you can revive the issue at any time.