Integration with kustomize SecretGenerator

[davinkevin]

Hello all,

I followed the evolution of https://github.com/GoogleCloudPlatform/k8s-config-connector/issues/27, but I found a limitation.

When using kubectl apply -k with a secret generator, it doesn't modify the name of the secretRef to include the hash of the file. This totally prevent from using kubectl apply -k with KCC when using secrets values.

For example:

dp.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sl-demo-app
spec:
  selector:
    matchLabels:
      app: sl-demo-app
  template:
    metadata:
      labels:
        app: sl-demo-app
    spec:
      containers:
      - name: app
        env:
        - name: "DB_PASSWORD"
          valueFrom:
            secretKeyRef:
              name: sl-demo-app
              key: db.password
        image: foo/bar:latest
        ports:
        - name: http
          containerPort: 8080
          protocol: TCP

sql.yaml

apiVersion: sql.cnrm.cloud.google.com/v1beta1
kind: SQLUser
metadata:
  name: foo
  labels:
    app: bar
spec:
  instanceRef:
    name: foo-001
  host: ""
  password:
    valueFrom:
      secretKeyRef:
        name: sl-demo-app
        key: db.password

Kustomization.yaml:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- dp.yaml
- sql.yaml

secretGenerator:
- literals:
  - db-password=12345
  name: sl-demo-app
  type: Opaque

If I generate it through kubectl apply -k . -o yaml (or with kustomize build .), I have the following result:

apiVersion: v1
items:
  - apiVersion: v1
    data:
      db-password: MTIzNDU=
    kind: Secret
    metadata:
      annotations:
        kubectl.kubernetes.io/last-applied-configuration: |
          {"apiVersion":"v1","data":{"db-password":"MTIzNDU="},"kind":"Secret","metadata":{"annotations":{},"name":"sl-demo-app-6ft88t2625","namespace":"subscription-management"},"type":"Opaque"}
      name: sl-demo-app-6ft88t2625
      namespace: subscription-management
    type: Opaque
  - apiVersion: apps/v1
    kind: Deployment
    metadata:
      annotations:
        kubectl.kubernetes.io/last-applied-configuration: |
          {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"name":"sl-demo-app","namespace":"subscription-management"},"spec":{"selector":{"matchLabels":{"app":"sl-demo-app"}},"template":{"metadata":{"labels":{"app":"sl-demo-app"}},"spec":{"containers":[{"env":[{"name":"DB_PASSWORD","valueFrom":{"secretKeyRef":{"key":"db.password","name":"sl-demo-app-6ft88t2625"}}}],"image":"foo/bar:latest","name":"app","ports":[{"containerPort":8080,"name":"http","protocol":"TCP"}]}]}}}}
      name: sl-demo-app
      namespace: subscription-management
    spec:
      selector:
        matchLabels:
          app: sl-demo-app
      template:
        metadata:
          labels:
            app: sl-demo-app
        spec:
          containers:
            - env:
                - name: DB_PASSWORD
                  valueFrom:
                    secretKeyRef:
                      key: db.password
                      name: sl-demo-app-6ft88t2625
              image: foo/bar:latest
              name: app
              ports:
                - containerPort: 8080
                  name: http
                  protocol: TCP
  - apiVersion: sql.cnrm.cloud.google.com/v1beta1
    kind: SQLUser
    metadata:
      annotations:
        kubectl.kubernetes.io/last-applied-configuration: |
          {"apiVersion":"sql.cnrm.cloud.google.com/v1beta1","kind":"SQLUser","metadata":{"annotations":{},"labels":{"app":"bar"},"name":"foo","namespace":"subscription-management"},"spec":{"host":"","instanceRef":{"name":"foo-001"},"password":{"valueFrom":{"secretKeyRef":{"key":"db.password","name":"sl-demo-app"}}}}}
      labels:
        app: bar
      name: foo
      namespace: subscription-management
    spec:
      host: ""
      instanceRef:
        name: foo-001
      password:
        valueFrom:
          secretKeyRef:
            key: db.password
            name: sl-demo-app
kind: List
metadata: {}

We see the name: sl-demo-app-6ft88t2625 has been injected into the deployment but not in the SQLUser.

Thanks for your help

[kibbles-n-bytes]

Hey @davinkevin ; just did a quick investigation of what this would entail. My understanding is that we would need to add the "x-kubernetes-object-ref-kind": "Secret" OpenAPI annotation to our CRDs' validation schemas, and then on the client side, your kustomization.yaml would need to refer to our CRDs via the crds field. Is this correct?

[davinkevin]

I think you're right but I'm not 100% sure. If you need someone to test something, let me know !

[davinkevin]

Up ?

Can I help on something on this ?

[maqiuyujoyce]

@davinkevin Thank you for the follow up. As @kibbles-n-bytes described above, a workaround for you would be adding "x-kubernetes-object-ref-kind": "Secret" OpenAPI annotation to the CRDs' validation schemas that your kustomization.yaml refers to via the crds field.

We are considering to support the feature and will update when we have more information.

[davinkevin]

When I try to use crds fields, I have the following error:

loading CRDs [kcc.crds.yaml]: unable to parse open API definition from 'kcc.crds.yaml': error unmarshaling JSON: while decoding JSON: json: cannot unmarshal string into Go value of type common.OpenAPIDefinition

[Neonox31]

Hey 👋

@davinkevin You can use a name reference transformer configuration like the following :

# sql-user-password-as-kustomize-secret.yaml
nameReference:
  - kind: Secret
    version: v1
    fieldSpecs:
      - path: spec/password/valueFrom/secretKeyRef/name
        kind: SQLUser

# kustomization.yaml
...
configurations:
  - sql-user-password-as-kustomize-secret.yaml
...

[davinkevin]

Thanks @Neonox31 ❤️

By the way, the best solution should be to update the KCC crds to support this annotation? Any news about this @kibbles-n-bytes ?

[giddel]

Hi, is there any progress in this issue? Would be nice if the config-connector can understand generated secrets!