Replacing existing/creation of secrets

[jetersen]

I base a basic setup:

| base
  | kustomization.yaml
  | cloudflare-api-token.yaml
  | secrets-generator.yaml
| cluster1
  | kustomization.yaml
  | cluster-specific-secret.yaml
  | secrets-generator.yaml
| clsuter2
  | kustomization.yaml
  | secrets-generator.yaml

When applying secrets for cluster1 that uses base I get the following error:

Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = `kustomize build <path to cached source>/staging --enable-alpha-plugins --enable-exec` failed exit status 1: Error: accumulating resources: accumulation err='accumulating resources from '../base': '<path to cached source>/base' must resolve to a file': recursed accumulation of path '<path to cached source>/base': merging from generator &{0xc0003cef00 <nil>}: id resid.ResId{Gvk:resid.Gvk{Group:"", Version:"v1", Kind:"Secret", isClusterScoped:false}, Name:"cert-manager-cloudflare", Namespace:"cert-manager"} does not exist; cannot merge or replace

Fails in argocd but also locally 🤔

[dosubot[bot]]

🤖

Hello @jetersen,

I'm here to assist you with your issue. I'm a bot designed to help solve bugs, answer questions, and guide you in becoming a contributor while you're waiting for a human maintainer.

Based on the error message you're encountering, it seems like the secret you're trying to apply does not exist in the cluster you're targeting. The replace behavior in the annotation kustomize.config.k8s.io/behavior: replace means that the resource will replace an existing resource in the cluster if one exists, but if it doesn't, it will fail with an error.

If you want to create a new resource if it doesn't exist and replace it if it does, you should use the create behavior instead. Here's how you can modify your secret:

apiVersion: v1
data:
    password: ENC[AES256_GCM,data:0fBeqB61eNd2EdiZpmm6Zg==,iv:NSzj5IKHZCaLx3NV2D9YXH0rqmo4DYltpORtlT0pDPg=,tag:VPgf4B6dAxvIn9YNaQ/Znw==,type:str]
    username: ENC[AES256_GCM,data:aIBt+MIxD/0=,iv:gHeevm4wbY0iuGt6MxeetGpPS15TXnRFBcOGZF2p+RE=,tag:4c2DqbB1QX+hE5a/DhnTWQ==,type:str]
kind: Secret
metadata:
    name: my-base-secret
    annotations:
        kustomize.config.k8s.io/behavior: create
stringData:
    application: ENC[AES256_GCM,data:tjy4jEUvcLBao9eFP/4=,iv:6rGjfBhi3LxH+MZ5Y8FJTmrFFUqCU/uYOTFOtN4/V4Q=,tag:UIs+ub7wqlv4MAytGLYGDg==,type:str]
type: Opaque
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age: []
    lastmodified: "2022-12-14T18:18:58Z"
    mac: ENC[AES256_GCM,data:XLyq51JJiz+9+uQkjB7snu7z9yoX50QW8q7y2ivO4E11wE7ab7womD0Ga3tYn8JR1wzDpVZ+F3434fR4tTaIt9WRZfKANsjtFBgOjhMJAQ1u7zcJiiDEID8VktUowK3p/pScEHy7Zsj2Mlf7Kpez0ZgeSzDgz1+aHquRb9Wj0mw=,iv:A/Th8d5C/3f2nVUxtsC70rWcpzIcGECvc9Qdm5S87rg=,tag:p1405TW+6zzEmcKrmojMjA==,type:str]
    pgp:
        - created_at: "2022-12-14T18:18:52Z"
          enc: |
            -----BEGIN PGP MESSAGE-----

            hQEMAyUpShfNkFB/AQgAm6fqwqD8HqVNcxYuOV2vEWKH2eANVCdOyhEP8g5BqFni
            BYOrWoIiZPGiH1XXLvpUC0b6ZbRM7Mi6CAhsLI5Msk+9mTwP/3uUCue+7fkodqg2
            CUnXPL3dVsaE6HtRnbq5eWajPOxyiPIWjjjZFSajJ9JN62d5Itq+KvdJlJuIsIay
            hTGYSpX1+fFOPk0Y7NxRi4dTaYPJbIrWVR+EDoIBt0QEYkKgTY5YrG0Yk8ovDeey
            iUDNIQFnhA+ci2hcMjvub1Nd9URPF8Zv+uMjE7fivG4/ORX5cVV137J5oSLhIUEk
            E4Z9j6EHdYF2a//Cnf2lzjjt3xslRgrsjClykzI1e9JeAZEFmQc6q73Qcgt5ZAK4
            g2iQ1CgXbbnvjBF49kHWshe2MQcFz3KyhSSDqgf4LAIV+znYfpTqvmo3WXvELuKx
            N36A0PwW1qmxSay9wVer4cvsTrNqBk77uJIaYpGv3w==
            =ER/2
            -----END PGP MESSAGE-----
          fp: FBC7B9E2A4F9289AC0C1D4843D16CEE4A27381B4
    unencrypted_regex: ^(apiVersion|metadata|kind|type)$
    version: 3.7.2

This should resolve the issue you're facing. If you have any other questions or need further clarification, feel free to ask.

Best, Dosu

Sources

• Replace existing secret?
• override helm secret
• test/krm/behaviors/secret.enc.yaml

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[jetersen]

A google search on "kustomize.config.k8s.io/behavior" does not provide any context for create vs replace

The readme links to https://github.com/kubernetes-sigs/kustomize/blob/master/examples/generatorOptions.md but this does not give insights into the behaviors that do exist 😓

[devstein]

Hey @jetersen the kustomize generator behaviors are poorly documented...I usually end up reading the kustomize code 🙁

This issue doesn't have to do with KSOPS, but is a quirk of the order in which kustomize evaluates your manifests. Are you able to share any more information about your manifests files? Hard to debug without seeing the manifests

[jetersen]

The behavior I was looking for was upsert and remove any unknown values. It seems the default is merging. I had a case with cert-manager-cloudflare having the key created manually with an token inside the secret data but ksops did not remove the unknown field.

base/kustomization.yaml

generators:
  - ./secrets-generator.yaml

base/secrets-generator.yaml

apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: secrets-generator
  annotations:
    config.kubernetes.io/function: |
        exec:
          path: ksops
files:
  - ./grafana.yaml
  - ./cloudflare-api-token.yaml

cloudflare-api-token.yaml

apiVersion: v1
kind: Secret
metadata:
    name: cert-manager-cloudflare
    namespace: cert-manager
    annotations:
        kustomize.config.k8s.io/behavior: replace
        kustomize.config.k8s.io/needs-hash: "true"
stringData:
    cloudflare_api_token: snip
type: Opaque
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age: []
    lastmodified: "2023-10-28T19:11:00Z"
    mac: snip
    pgp: []
    encrypted_regex: ^(data|stringData)$
    version: 3.8.0
---
apiVersion: v1
kind: Secret
metadata:
    name: external-dns-cloudflare
    namespace: external-dns
    annotations:
        kustomize.config.k8s.io/behavior: replace
        kustomize.config.k8s.io/needs-hash: "true"
stringData:
    cloudflare_api_token: snip
type: Opaque
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    hc_vault: []
    age: []
    lastmodified: "2023-10-28T19:11:00Z"
    mac: snip
    pgp: []
    encrypted_regex: ^(data|stringData)$
    version: 3.8.0