Custom YAML Tags

[DreamingRaven]

I have an issue where I want to support custom YAML tags for a library I am creating an operator for (https://gitlab.com/GeorgeRaven/authentik-manager).

To do so I created a custom type so I could handle the custom logic required for these yaml tags like !Find:

apiVersion: akm.goauthentik.io/v1alpha1
kind: AkBlueprint
metadata:
  labels:
    app.kubernetes.io/name: akblueprint
    app.kubernetes.io/instance: akm
    app.kubernetes.io/part-of: operator
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: operator
  name: tenant
  namespace: auth
spec:
  file: /blueprints/default/default-tenant.yaml
  blueprint:
    metadata:
      name: Default - Tenant
    version: 1
    entries:
    - model: authentik_blueprints.metaapplyblueprint
      attrs:
        identifiers:
          name: Default - Authentication flow
        required: false
    - model: authentik_blueprints.metaapplyblueprint
      attrs:
        identifiers:
          name: Default - Invalidation flow
        required: false
    - model: authentik_blueprints.metaapplyblueprint
      attrs:
        identifiers:
          name: Default - User settings flow
        required: false
    - attrs:
        flow_authentication: !Find [authentik_flows.flow, [slug, default-authentication-flow]]
        flow_invalidation: !Find [authentik_flows.flow, [slug, default-invalidation-flow]]
        flow_user_settings: !Find [authentik_flows.flow, [slug, default-user-settings-flow]]
        branding_title: YourBranding
      identifiers:
        domain: authentik-default
        default: True
      state: create
      model: authentik_tenants.tenant

My custom type is called Raw and its only purpose is to maintain the yaml tags as strings so that they can be given to authentik. To do so I base Raw on yaml.v3 nodes since these allow finer grained control to handle these tags which AFAIK yaml.v2 which the k8s forks based on https://github.com/ghodss/yaml do not.

Unmarshalling and marshalling of my API type works almost as expected with yaml.v3 funcs as you'd expect since it's based off of yaml.Nodes (minus a subslice that I need to add support for, and the k8s specific object meta):

func TestAkBpApiYaml(t *testing.T) {
    bytes := exampleAkBlueprintYaml()
    fmt.Printf("%s\n", bytes)
    akbp := &AkBlueprint{}
    err := yaml_v3.Unmarshal(bytes, akbp)
    if err != nil {
        t.Fatalf("Failed to unmarshal YAML: %v", err)
    }
    fmt.Printf("%+v\n", akbp)
    remarshalled, err := yaml_v3.Marshal(akbp)
    if err != nil {
        t.Fatalf("Failed to marshal YAML: %v", err)
    }
    fmt.Printf("%s\n", remarshalled)
        ...
}

produces:

kind: AkBlueprint
apiversion: ""
metadata:
    name: tenant
    generatename: ""
    namespace: auth
    selflink: ""
    uid: ""
    resourceversion: ""
    generation: 0
    creationtimestamp: "0001-01-01T00:00:00Z"
    deletiontimestamp: null
    deletiongraceperiodseconds: null
    labels:
        app.kubernetes.io/created-by: operator
        app.kubernetes.io/instance: akm
        app.kubernetes.io/managed-by: kustomize
        app.kubernetes.io/name: akblueprint
        app.kubernetes.io/part-of: operator
    annotations: {}
    ownerreferences: []
    finalizers: []
    managedfields: []
spec:
    file: /blueprints/default/default-tenant.yaml
    blueprint:
        version: 1
        metadata:
            name: Default - Tenant
        entries:
            - model: authentik_blueprints.metaapplyblueprint
              attrs:
                identifiers:
                    name: Default - Authentication flow
                required: "false"
            - model: authentik_blueprints.metaapplyblueprint
              attrs:
                identifiers:
                    name: Default - Invalidation flow
                required: "false"
            - model: authentik_blueprints.metaapplyblueprint
              attrs:
                identifiers:
                    name: Default - User settings flow
                required: "false"
            - model: authentik_tenants.tenant
              state: create
              identifiers:
                default: "True"
                domain: authentik-default
              attrs:
                branding_title: YourBranding
                flow_authentication: '!Find [authentik_flows.flow ]'
                flow_invalidation: '!Find [authentik_flows.flow ]'
                flow_user_settings: '!Find [authentik_flows.flow ]'

However using the k8s sigs yaml "sigs.k8s.io/yaml" used in the kubebuilder controller (mocked in tests):

func TestAkBpApiK8sYaml(t *testing.T) {
    bytes := exampleAkBlueprintYaml()
    fmt.Printf("%s\n", bytes)
    akbp := &AkBlueprint{}
    err := yaml_k8s.Unmarshal(bytes, akbp)
    if err != nil {
        t.Fatalf("Failed to unmarshal YAML: %v", err)
    }
    fmt.Printf("%+v\n", akbp)
    remarshalled, err := yaml_k8s.Marshal(akbp)
    if err != nil {
        t.Fatalf("Failed to marshal YAML: %v", err)
    }
    fmt.Printf("%s\n", remarshalled)
    t.Fatalf("Not implemented")
}

Results in losing the !Find custom yaml tags:

apiVersion: akm.goauthentik.io/v1alpha1
kind: AkBlueprint
metadata:
  creationTimestamp: null
  labels:
    app.kubernetes.io/created-by: operator
    app.kubernetes.io/instance: akm
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/name: akblueprint
    app.kubernetes.io/part-of: operator
  name: tenant
  namespace: auth
spec:
  blueprint:
    entries:
    - attrs:
        identifiers:
          name: Default - Authentication flow
        required: "false"
      model: authentik_blueprints.metaapplyblueprint
    - attrs:
        identifiers:
          name: Default - Invalidation flow
        required: "false"
      model: authentik_blueprints.metaapplyblueprint
    - attrs:
        identifiers:
          name: Default - User settings flow
        required: "false"
      model: authentik_blueprints.metaapplyblueprint
    - attrs:
        branding_title: YourBranding
        flow_authentication:
        - authentik_flows.flow
        - - slug
          - default-authentication-flow
        flow_invalidation:
        - authentik_flows.flow
        - - slug
          - default-invalidation-flow
        flow_user_settings:
        - authentik_flows.flow
        - - slug
          - default-user-settings-flow
      identifiers:
        default: "true"
        domain: authentik-default
      model: authentik_tenants.tenant
      state: create
    metadata:
      name: Default - Tenant
    version: 1
  file: /blueprints/default/default-tenant.yaml
status: {}

What is the best way to incorporate the yaml.v3 node functionality in a controller-runtime based controller so that I can handle custom Yaml tags? Thanks, any information would be much appreciated!

Edit: I realise there may be a lot of context missing here but it is quite a multi-level problem as I see it, if you would like any information on specifics please let me know, but broadly I want to handle Yaml tags myself or by go-yaml v3 without having the controller runtime mutate them on Get.

[troy0820]

/kind support

[alvaroaleman]

Controller-Runtime does not support using yaml encoding when communicating with the k8s apiserver, it uses protobuf or json in the case of CRDs.

How you do or do not serialize your types outside of controller-runtime is your decision. Given that json is valid yaml however, I strongly recommend to not use custom yaml de/encoding logic, as it will silently do the wrong thing when reading a json-serialized object.

[DreamingRaven]

Thanks for following up, unfortunately I have had to do just this. I now have to treat the blueprints from authentik as multi line strings to avoid them being mutated outside of my code since they require these custom yaml tags as is much to my chagrin. I wish it wasn't the case unfortunately, handling custom yaml tags in go is a bit of a nightmare.

I wanted to properly unmarshal into non strings to provide more structure to end users but as you say it fails silently. Thanks again for the follow up.