Attention: BREAKING Webhook secret label in 0.10.0

[Skarlso]

TL;DR

Any Secrets Controlled (Created, Updated, Deleted) by ExternalSecrets are not affected. You don't need to do anything

If you use Webhook Provider with templating - you need to update any Secrets that are source of your template values.

This issue is a placeholder should anyone encounter a problem regarding the webhook and its secret.

BREAKING CHANGE:

• Webhook Generator Webhook generator labels have changed from generators.external-secrets.io/type: webhook to external-secrets.io/type: webhook.

• Webhook Provider Webhook provider now can only use secrets that are labeled with external-secrets.io/type: webhook. This enforces explicit setup for webhook secrets by users.

Fixing the issue:

add the label for the secret used by the webhook:

apiVersion: v1
kind: Secret
metadata:
  name: your-secret
  labels:
    external-secrets.io/type: webhook ### <<<<<<<<<<<<< ADD THIS
data:
...

Pull Request: https://github.com/external-secrets/external-secrets/pull/3753

[frantathefranta]

How should this be added to an ExternalSecret CRD? I've tried adding it in the top-level and for the generated secret but neither worked:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: &name minio
  namespace: storage
  labels:
    external-secrets.io/type: webhook
spec:
  secretStoreRef:
    name: bitwarden-secrets-manager
    kind: ClusterSecretStore
  target:
    name: *name
    template:
      engineVersion: v2
      metadata:
        labels:
          external-secrets.io/type: webhook

[brunnels]

@Skarlso The new label isn't working. I had to rollback the version.

1. First I added the label to all my secrets by adding the template metadata like @frantathefranta has above
2. Then I verified all my secrets created via ExternalSecret had the new label. k get secrets -A -l external-secrets.io/type=webhook
3. Next I updated to the version 0.10.0
4. Finally I deleted one of my externalsecrets and then had flux reconcile recreate it.

I see the error when I describe the externalsecret: Warning UpdateFailed 4s (x13 over 24s) external-secrets secret does not contain needed label 'external-secrets.io/type: webhook'. Update secret label to use it with webhook

[Skarlso]

It's not the external secrets that needs this label. It's the secret that the Webhook is allowed use define in here:

    // Secrets to fill in templates
    // These secrets will be passed to the templating function as key value pairs under the given name
    // +optional
    Secrets []Secret `json:"secrets,omitempty"`

For example, in case of the generator like this:

apiVersion: generators.external-secrets.io/v1alpha1
kind: Webhook
metadata:
  name: webhook
spec:
  url: "http://httpbin.org/get?parameter={{ .auth.param }}"
  result:
    jsonPath: "$.args"
  headers:
    Content-Type: application/json
    Authorization: Basic {{ print .auth.username ":" .auth.password | b64enc }}
  secrets:
  - name: auth
    secretRef:
      name: webhook-credentials
---
apiVersion: v1
kind: Secret
metadata:
  name: webhook-credentials
  labels:
    external-secrets.io/type: webhook #Needed to allow webhook to use this secret
data:
  username: dGVzdA== # "test"
  password: dGVzdA== # "test"
  param: dGVzdA== # "test"

[Skarlso]

Oh sorry, you're saying that you use ExternalSecret to generate the Secret that webhook will than use and that the label created by ExternalSecret isn't copied over to the new secret?

[Skarlso]

I tried it but seems like it's working:

These are the stores and ES:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: secret-store-name
  namespace: default
spec:
  provider:
    kubernetes:
      auth:
        serviceAccount:
          # This would be the project service account which has the right permissions to create secrets.
          name: "default"
      server:
        caProvider: # we are connecting to our own cluster.
          type: ConfigMap
          name: kube-root-ca.crt
          key: ca.crt
---
apiVersion: external-secrets.io/v1beta1 
kind: ExternalSecret
metadata:
  name: find-by-name
  labels:
    external-secrets.io/type: webhook
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: secret-store-name
    kind: SecretStore
  target:
    name: secret-to-be-created
  dataFrom:
  - find:
      name:
        regexp: "git-*"

And the generated secret:

apiVersion: v1
data:
  ...
kind: Secret
metadata:
  ...
  labels:
    reconcile.external-secrets.io/created-by: c02f217d416ce4c3489a545e1e96b2f9
    external-secrets.io/type: webhook
  name: secret-to-be-created
  namespace: default
type: Opaque

[nevenamaksic1]

I don't think it's clear which k8s resources should have the annotation added before the upgrade. Should we add the annotation only to the secret that the webhook CRD is using, or should it (also) be added somewhere else?

[Skarlso]

Well, the description is saying this:

Webhook Generator Webhook generator labels have changed from generators.external-secrets.io/type: webhook to external-secrets.io/type: webhook.

Webhook Provider Webhook provider now can only use secrets that are labeled with external-secrets.io/type: webhook. This enforces explicit setup for webhook secrets by users.

So only the webhook provider or generator.

And then further:

add the label for the secret used by the webhook:

This defines which secret is being talked about.

If you have some better words / definition, please feel free to provide them I'll update the issue description if it's clearer. :)

[nevenamaksic1]

Sorry for the confusion, the comments from this thread made things a bit unclear for me. Thank you for the further explanation :)

[Skarlso]

No worries. :)

[brunnels]

I tried it but seems like it's working:

These are the stores and ES:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: secret-store-name
  namespace: default
spec:
  provider:
    kubernetes:
      auth:
        serviceAccount:
          # This would be the project service account which has the right permissions to create secrets.
          name: "default"
      server:
        caProvider: # we are connecting to our own cluster.
          type: ConfigMap
          name: kube-root-ca.crt
          key: ca.crt
---
apiVersion: external-secrets.io/v1beta1 
kind: ExternalSecret
metadata:
  name: find-by-name
  labels:
    external-secrets.io/type: webhook
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: secret-store-name
    kind: SecretStore
  target:
    name: secret-to-be-created
  dataFrom:
  - find:
      name:
        regexp: "git-*"

And the generated secret:

apiVersion: v1
data:
  ...
kind: Secret
metadata:
  ...
  labels:
    reconcile.external-secrets.io/created-by: c02f217d416ce4c3489a545e1e96b2f9
    external-secrets.io/type: webhook
  name: secret-to-be-created
  namespace: default
type: Opaque

@Skarlso This is what @frantathefranta reported above as not working except that he's using ClusterSecretStore. Did it work for you using a ClusterSecretStore?

[Skarlso]

I'll try, but it shouldn't matter, since the external-secrets-controller loop is the one propagating the labels.

[joryirving]

I'm curious why this is needed, as I currently don't add any labels to my external secrets, so this is a strange new addition that I would expect ESO to add automatically.

[gusfcarvalho]

TL;DR

Any Secrets Controlled (Created, Updated, Deleted) by ExternalSecrets are not affected. You don't need to do anything

If you use Webhook Provider with templating - you need to update any Secrets that are source of your template values.

Complete

This update was to be sure only explicit secrets can be used by webhook. It means it is for users to add this annotation to any kubernetes Secret used by webhook templating.

In this case:

apiVersion: generators.external-secrets.io/v1alpha1
kind: Webhook
metadata:
  name: webhook
spec:
  url: "http://httpbin.org/get?parameter={{ .auth.param }}"
  result:
    jsonPath: "$.args"
  headers:
    Content-Type: application/json
    Authorization: Basic {{ print .auth.username ":" .auth.password | b64enc }}
  secrets:
  - name: auth
    secretRef:
      name: webhook-credentials

webhook-credentials secret need to have the label. Otherwise webhook provider will fail.

For any other case, you don't need to do anything, as this breaking change is exclusive to webhook provider.

[gusfcarvalho]

@frantathefranta @brunnels @joryirving @nevenamaksic1 does this clarify your questions? Are you seeing any issues related to this breaking change?

Btw, thanks for the patience 😄

[joryirving]

That cleared it up. I only needed to add the label to the secret I'm passing for credentials. It's now working as expected.

[frantathefranta]

Works for me now as well. Looking back at it, makes sense how you've explained it, I just didn't realize how it was supposed to come together.

[brunnels]

@Skarlso Thanks for the assist. That resolved it for me as well