Strimzi operator removes annotations from secrets with custom credentials

[jharting]

Describe the bug A way to define custom credentials for a KafkaUser is to define define the credentials in a secret with a matching name and labeled with strimzi.io/cluster and strimzi.io/kind labels, for example:

apiVersion: v1
kind: Secret
metadata:
  name: cyndi
  labels:
    strimzi.io/cluster: my-cluster
    strimzi.io/kind: KafkaUser
  annotations:
    foo: bar
data:
  password: aGF2ZWZ1bg==
  sasl.jaas.config: b3JnLmFwYWNoZS5rYWZrYS5jb21tb24uc2VjdXJpdHkuc2NyYW0uU2NyYW1Mb2dpbk1vZHVsZSByZXF1aXJlZCB1c2VybmFtZT0iY3luZGkiIHBhc3N3b3JkPSJoYXZlZnVuIjs=
type: Opaque

The strimzi operator then uses these credentials for the user instead of generating new ones, which is great.

However, during the reconciliation, the operator also modifies the secret and removes all the annotations. This is not such a problem in the foo: bar scenario but it becomes a problem when a gitops tools is used to provision (and rotate) the credentials in the secret. These tools typically make use of annotations and get confused or work unexpectedly if the strimzi operator interferes with the annotations they use.

To Reproduce Steps to reproduce the behavior:

1. Deploy Strimzi/AMQ Streams (I am using amqstreams.v1.7.2)
2. Create a custom secret with kafka user credentials (see above)
3. Create a KafkaUser with the matching name
4. Observe the strimzi-operator take ownership of the secret and remove all annotations

Expected behavior The operator should read the provided credentials (and add custom labels/annotations if needed) but should otherwise respect the state of the resource and should not touch labels/annotations it had not set.

Environment (please complete the following information):

• Strimzi version: amqstreams.v1.7.2
• Installation method: OLM
• Kubernetes cluster: CodeReady Containers version: 1.26.0+31f06c09
• Infrastructure: Run locally on my Fedora laptop

[scholzj]

Right now, this is works as designed. The labels and annotations on the secret are controlled by Strimzi who owns it (although in your case you actually create it manually first). To add additional annotations or labels, you can use the template section of the KafkaUser resource: https://strimzi.io/docs/operators/0.22.0/full/using.html#type-KafkaUserTemplate-reference ... not sure how your GitOps tool uses the annotation to rotate the secret, but maybe this helps?

[jharting]

To add additional annotations or labels, you can use the template section of the KafkaUser resource: https://strimzi.io/docs/operators/0.22.0/full/using.html#type-KafkaUserTemplate-reference ... not sure how your GitOps tool uses the annotation to rotate the secret, but maybe this helps?

This won't unfortunately help me as the labels are dynamic.

Right now, this is works as designed. The labels and annotations on the secret are controlled by Strimzi who owns it (although in your case you actually create it manually first).

I think it makes sense for the operator to own the secret with auto-generated credentials. However, if I am supplying custom secret then this looks odd. Why does the operator need to have ownership of the secret that was provided to it? I think it would make sense for the operator to read the provided credentials and watch the secret for changes but without redefining it.

I think this is going to be a common problem with gitops-like tools that manage kubernetes resources based on a central source of truth. Either the gitops tool manages the secret (e.g. based on an entry in Vault) or the strimzi-operator manages it (ideally only when a custom one is not provided). Otherwise, the two actors will run into conflicts.

Perhaps there should be a different way to specify custom credentials that is clearly separated from where Strimzi stores the credentials internal i.e. the Strimzi-managed Secret instance (e.g. an explicit reference to a Secret in KafkaUserSpec)

[scholzj]

I think it makes sense for the operator to own the secret with auto-generated credentials. However, if I am supplying custom secret then this looks odd. Why does the operator need to have ownership of the secret that was provided to it? I think it would make sense for the operator to read the provided credentials and watch the secret for changes but without redefining it.

Well, you are not providing the secret to the operator. You are tricking it into thinking it already created the secret in the past with the password you want. So the operator still sees the ownership there. So from my point of view this works as expected. In order to achieve what you want, we would need to add actual support for providing your own secret with the password which will be linked in the KafkaUser resource and the operator will read the password from it and use it to create its own secret (as you suggested as well). This would need to be added as a new feature.

I think this is going to be a common problem with gitops-like tools that manage kubernetes resources based on a central source of truth. Either the gitops tool manages the secret (e.g. based on an entry in Vault) or the strimzi-operator manages it (ideally only when a custom one is not provided). Otherwise, the two actors will run into conflicts.

It does not seem so - there were many users in the past who wanted to set their own passwords, but I do not think anyone is using GitOps to set passwords like this. That seems a bit like an anti-pattern to me. Why store password in SCM which will have always some security risk when you can have them auto-generated directly in Kubernetes? (Having the link to some secret with the password in the KafkaUser resource might make sense also for other reasons, so it does not necessarily matter that much what I think about the pattern)

You mentioned Vault as well ... the question is, why do you want to use Vault and if you really want the passwords to be stored in Vault and also in the Kubernetes Secrets? So another option might be to add some support for the User Operator to directly use Vault to store or read the passwords. I do not really have any experience with Vault, but different Vault enhancements seem to be quite common. So it might be worth investigating that as well if your primary target is Vault and not GitOps.

[jharting]

Well, you are not providing the secret to the operator. You are tricking it into thinking it already created the secret in the past with the password you want. So the operator still sees the ownership there. So from my point of view this works as expected.

Fair enough. I think I had assumed that what I was doing was the official way to provide custom credentials but now I understand that it's sort of a gray area where the goal can be achieved but there are caveats.

In order to achieve what you want, we would need to add actual support for providing your own secret with the password which will be linked in the KafkaUser resource and the operator will read the password from it and use it to create its own secret (as you suggested as well). This would need to be added as a new feature.

OK, that sounds great. Do you want me to open a new feature request issue or do you want to repurpose this one?

[scholzj]

Maybe a new one with a clearer description might be good. Will you create one? Or you want me to do it?