Using transformations in hull.objects.secret.*.data.*.inline

Hello again!

I'm sorry for the disturbance; there's another question if you please.

Here is a minimalistic values.yaml:

hull:
  objects:
    secret:
      my-secret-1:
        data:
          .dockerconfigjson:
            inline: _HT/myUtilities.generateDotDockerConfigJson
      my-secret-2:
        data:
          .dockerconfigjson:
            inline: _HT!{{ include "myUtilities.generateDotDockerConfigJson" . }}

And here's the file with myUtilities.generateDotDockerConfigJson:

{{ define "myUtilities.generateDotDockerConfigJson" }}
{"auths":{"my-registry":{"username":"username","password":"password","email":"email","auth":"dXNlcm5hbWU6cGFzc3dvcmQ="}}}
{{ end }}

Everything seems to be quite straightforward, but in my-secret-1 I'm getting bWFwW2F1dGhzOm1hcFtteS1yZWdpc3RyeTptYXBbYXV0aDpkWE5sY201aGJXVTZjR0Z6YzNkdmNtUT0gZW1haWw6ZW1haWwgcGFzc3dvcmQ6cGFzc3dvcmQgdXNlcm5hbWU6dXNlcm5hbWVdXV0= (which means map[auths:map[my-registry:map[auth:dXNlcm5hbWU6cGFzc3dvcmQ= email:email password:password username:username]]]) and in my-secret-2 I'm getting just "".

It seems that there's a special handling for hull.objects.secret.*.data.*.inline when it looks like a JSON-structure, but I can't find out how to pass the result of myUtilities.generateDotDockerConfigJson as a string. Would you be so kind as to help me out again?

Thank you very much in advance.

I rewrote my-secret-2 in the following way:

      my-secret-2:
        data:
          .dockerconfigjson:
            inline: _HT!{{ include "myUtilities.generateDotDockerConfigJson" . | toJson }}

Now it works fine, it gives eyJhdXRocyI6eyJteS1yZWdpc3RyeSI6eyJ1c2VybmFtZSI6InVzZXJuYW1lIiwicGFzc3dvcmQiOiJwYXNzd29yZCIsImVtYWlsIjoiZW1haWwiLCJhdXRoIjoiZFhObGNtNWhiV1U2Y0dGemMzZHZjbVE9In19fQ==, which means exactly what I expected to see ({"auths":{"my-registry":{"username":"username","password":"password","email":"email","auth":"dXNlcm5hbWU6cGFzc3dvcmQ="}}}).

I would like to clarify 2 things if you don't mind.

Do I understand it right that all JSON strings are being "fromJsonized" by the the hull.util.transformation.tpl transformations? So, if there's something that looks like JSON, hull.util.transformation.tpl will unmarshall it, am I right?
Why doesn't the hull.util.transformation.include work in my-secret-1, what do I do wrong?

Again, thank you for your time, attention and patience. And for the great project you shared, of course. :-)

I decided to make sure that unmarshalling is not being performed by Helm itself, so I've made a template (templates/secret.json) to check if the include function would "deJSONify" the value by itself.

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
data:
  my-value: {{ include "myUtilities.generateDotDockerConfigJson" . }}

It gave me a correct value: {"auths":{"my-registry":{"username":"username","password":"password","email":"email","auth":"dXNlcm5hbWU6cGFzc3dvcmQ="}}}.

Of course, the data must be base64-encoded in a secret, so it's not a valid secret and I should add | b64enc to this template, but the fact is a fact: the include doesn't unmarshall JSON by itself.

Hello again @khmarochos,

First of, thank you very much for this report. Sorry for late reply, it took me quite some time to analyze but it brought forth interesting insights into what may go wrong and what could be improved. Much appreciated!

I will try to present some background again and in the next post give some thoughts on improvements that could come out of this analysis.

In terms of ConfigMap and Secret management, one of the original ideas for HULL was to make it as easy and "natural feeling" as possible to add content inline in values.yaml and use templating in the content. This includes the possibility to use templating in the content and auto-base64 endoding for secrets. To achieve this, in the HULL code tpl is by default always run on inline content (or on the content in an external file if using the path property instead). The original chart context . or $ is passed to this tpl call and can be accessed directly as in regular Helm chart templates.

This is unlike (bascially all) other properties in the HULL YAML where you need to use the _HT//_HT! transformations to execute an include or do something else on which tpl is run, Technically the _HT is performed very differently from the straight-forward inline handling.

Given that, your valid solution:

inline: _HT!{{ include "myUtilities.generateDotDockerConfigJson" . | toJson }}

can even be reduced to directly doing the include and converting toJson:

inline: |-
  {{ include "myUtilities.generateDotDockerConfigJson" . | toJson }}

inline: '{{ include "myUtilities.generateDotDockerConfigJson" . | toJson }}'

and it should give you the same result. You could also do a toPrettyJson and it looks nicer in the output if you want.

I do think I need to highlight this more in the documentation but when specifying the content of ConfigMaps and Secrets this means you:

do not need to use _HT! or _HT/ to be able to use templating because tpl is always executed automatically
have the original contexts . and $ at your disposal and can access them in the inline content directly
can also access . and $ directly when you use an external content from a file via path, the intended behavior is that inline or external path content is treated the exact same way.

But the question remains, what goes wrong here with the `_HT/` wrapped `include`?

A little detour:

Functions are really powerful because they can return complex obejcts such as dicts and lists. A function result can provide a complete sub tree of YAML. In our additional library for company use we combine with HULL we have defined functions for creating many recurring structures in our Helm charts and can efficiently wrap big parts of recurring object creation into short _HT/ calls. Some real life examples from the library usage:

hull:
  objects:
    configmap:
      deletion-observer-api:
        data: _HT/hull.vidispine.addon.library.component.configmap.data:COMPONENT:(index . "OBJECT_INSTANCE_KEY")
    ingress:
      deletion-observer-api:
        rules: _HT/hull.vidispine.addon.library.component.ingress.rules:COMPONENTS:"deletion-observer-api,legacy-deletion-observer-api":SERVICENAME:(index . "OBJECT_INSTANCE_KEY")

This pays off of course only when you have many structurally similar objects in multiple Helm charts. I mostly want to highlight that _HT/ can return rather complex objects that are inserted into the YAML tree.

There are at least three ways how you can define complex objects (dict or list) to return from a function:

in Go templating code (which should be converted toYaml string in the end)
in usual block style YAML
in flow style YAML

Method 1. is probably cleanest since you don't have to deal with indentation at all and it can be applied everywhere without issues essentially. You just need to do a toYaml when returning it.

Method 2. and 3. can be more problematic because as you saw it depends on the correct embedding of the function call - if the literal result is not placed carefully it can break the YAML structure. Flow style YAML is still more forgiving in the sense that it doesn't rely as much on precise indentation as block style YAML does.

To highlight this, all of the three functions below return essentially the same valid YAML which may be converted to a dictionary by executing fromYaml on the result:

# block style YAML
{{- define "hull.include.test.imagepullsecrets.nonemptyarray" -}}
result:
- name: "a"
- name: "b"
{{- end -}}

# flow style YAML
{{- define "hull.include.test.imagepullsecrets.nonemptyflow" -}}
{ "result": [ { "name": "flowa" }, { "name": "flowb" } ] }
{{- end -}}

# code toYaml
{{- define "hull.include.test.imagepullsecrets.nonemptylist" -}}
{{ dict "result" (list (dict "name" "listreg1") (dict "name" "listreg2")) | toYaml }}
{{- end -}}

You'll notice that the second variant is pretty much the same as your myUtilities.generateDotDockerConfigJson function structurally. Only difference between flow style YAML and JSON is that in flow style YAML you don't need to quote strings but this means that any JSON string result of an include will be treated as a flow style YAML object. And this is kind of the problem that comes with this code:

.dockerconfigjson:
  inline: _HT/myUtilities.generateDotDockerConfigJson

where your function internally returns a dict object (even though from your intention it is a JSON string). The code in _HT/ must do a fromYaml to get a dict object which holds the data that is inserted into the YAML tree (like in the example usages of _HT/ presented above). Having this include result converted to a dict object is what we want and need at this point.

The final aspect that plays into this is the fact that there is a subsequent toString executed on the dict object and the outcome is then the map[...] which is the serialized dict. There are only a few places in the code which always do a toString conversion on anything you put into the value because in the output the type must be string on the Kubernetes side. This is very helpful for example since it allows you to put in or refrerence for example integers and they will safely be converted to string on rendering - but the downside is that if the input is a dict or list we get an undesirable representation of the object as a string.

toString is executed on the inline properties for ConfigMaps and Secrets and the values of annotations and labels only so this is where the unwanted map[...] representation may show up.

To summarize:

The problem you brought to light comes up due to a combination of aspects:

your _HT/ result technically is interpreted as a dict in HULL. Even if you want it to be treated as a string, there is now way to avoid this conversion because the JSON character sequence returned from the include will always be interpreted as a flow style YAML object and be converted to a dict/map.
forcing the dict toString creates the unwanted serilaized object representation
you can overcome the problem by not using _HT/ in the inline and do the include and toJson directly

As a consequence, it is:

generally not possible to use an _HT/ to produce a dict/list and store its JSON/YAML representation in an annotation or label because the internal conversion will mess up the result.
For inline there is a cleaner path as explained to use include and do what you want with the result. Nevertheless I realized there is some nice potential in improving the ConfigMap and Secret inline handling in terms of coping with JSON/YAML treatment. I want to offer possibilities auto-convert dict/list input to correct YAML/JSON string output.

My idea on how to improve this will be subject of the following post. Sorry that it may be a lot to take in but as mentioned it was quite hard to analyze yet very insightful.

The following related features I think could be helpful:

Allow optional string conversion method in _HT/

When using _HT/ an optional serialization instruction can be included in case it is desired to represent dicts/lists as strings. This may be appended using | after the include name:

_HT/my.include|toJson:VAR1:"var1":VAR2:"var2" _HT/key/my.include|toJson:VAR1:"var1":VAR2:"var2"

It is possible to add one of the following values:
- toJson
- toPrettyJson
- toRawJson
- toYaml
- toString
Technically the fromYaml conversion takes place but the result is then serialized accordingly and will be a string in the output. This would allow to pack dict/list content into an annotation or label in the wanted serialized form. This is possible using _HT! already but this would be way more convenient to use a brief infix here.
ConfigMap and Secret include handling

(In a less general way this already exists in the https://github.com/vidispine/hull-vidispine-addon but it may be generalized to be an optional part of HULL functionality)

The following functionality may be added (can be opted out in the `hull.config.general'):
- allow dict or list input to inline field (schema change). This could come from:
  - direct input like:
```
inline:
a:
b: "abc"
c: 
  d: 123
```
  - result of an include via _HT!
  - a reference via _HT*
- when the filename/object key in data ends with .json or .yaml or .yml the dict/list is autoconverted to JSON/YAML
- it is possible to use new key conversionMethod next to inline where you can determine the conversion that should happen on dict/list input to inline. This overrides automatic file ending conversions and is required for dict/list input when none of the auto-detectable endings is present

@Baum053 @khmarochos What do you think of the features?

Ah, now I understood how it happens, thank you!

I need some time to walk around this topic and then I'll be ready share some thoughts.

For 1. I did implement it in a first draft and I think it is going to be a nice feature.

Did decide to put any optional serialization instruction (I will call it "serialization" from now on) in front of the transformations value and separate with |. For now it can be the list of values from above:

toJson
toPrettyJson
toRawJson
toYaml
toString

If used as optional prefix in front it makes it more applicable to any transformations since some allow parameters at the end and it could mess with the parameter values.

So for _HT/ and _HT* you can then do this:

_HT/toJson|hull.include.test.dockerconfigjson.code
_HT*toJson|hull.config.specific.pod_spec
_HT/toPrettyJson|hull.include.test.dockerconfigjson.code
_HT*toPrettyJson|hull.config.specific.pod_spec
...

and any dict/list result from the _HT/ or _HT*will be serialized in the respective format.

So for this example input with your function:

{{- define "hull.include.test.dockerconfigjson.code" -}}
{{- $reg := dict "my-registry" (dict "username" "username" "password" "password" "email" "email" "auth" "dXNlcm5hbWU6cGFzc3dvcmQ=") -}}
{{- $auths := dict "auths" $reg -}}
{{- $auths | toYaml -}}
{{- end -}}

and values.yaml

hull:
  config:
    specific:
      pod_spec:
        initContainers:
          init:
            args:
            - or
            - use
            - this
            image:
              repository: extreg
              tag: youngest
            livenessProbe:
              initialDelaySeconds: 21
              periodSeconds: 22
              failureThreshold: 23
              timeoutSeconds: 24
              httpGet:
                path: /route
                scheme: HTTP
                port: 876
  objects:
    configmap:
      test-serializing:
        annotations: 
          test-include-code: _HT/hull.include.test.dockerconfigjson.code
          test-include-code-json: _HT/toJson|hull.include.test.dockerconfigjson.code
          test-include-code-prettyjson: _HT/toPrettyJson|hull.include.test.dockerconfigjson.code
          test-include-code-rawjson: _HT/toRawJson|hull.include.test.dockerconfigjson.code
          test-include-code-yaml: _HT/toYaml|hull.include.test.dockerconfigjson.code
          test-include-code-string: _HT/toString|hull.include.test.dockerconfigjson.code
          test-get: _HT*hull.config.specific.pod_spec
          test-get-json: _HT*toJson|hull.config.specific.pod_spec
          test-get-prettyjson: _HT*toPrettyJson|hull.config.specific.pod_spec
          test-get-rawjson: _HT*toRawJson|hull.config.specific.pod_spec
          test-get-yaml: _HT*toYaml|hull.config.specific.pod_spec
          test-get-string: _HT*toString|hull.config.specific.pod_spec
        data:
          test-include-code: 
            inline: _HT/hull.include.test.dockerconfigjson.code
          test-include-code-json: 
            inline: _HT/toJson|hull.include.test.dockerconfigjson.code
          test-include-code-prettyjson: 
            inline: _HT/toPrettyJson|hull.include.test.dockerconfigjson.code
          test-include-code-rawjson: 
            inline: _HT/toRawJson|hull.include.test.dockerconfigjson.code
          test-include-code-yaml: 
            inline: _HT/toYaml|hull.include.test.dockerconfigjson.code
          test-include-code-string: 
            inline: _HT/toString|hull.include.test.dockerconfigjson.code
          test-get: 
            inline: _HT*hull.config.specific.pod_spec
          test-get-json: 
            inline: _HT*toJson|hull.config.specific.pod_spec
          test-get-prettyjson: 
            inline: _HT*toPrettyJson|hull.config.specific.pod_spec
          test-get-rawjson: 
            inline: _HT*toRawJson|hull.config.specific.pod_spec
          test-get-yaml: 
            inline: _HT*toYaml|hull.config.specific.pod_spec
          test-get-string: 
            inline: _HT*toString|hull.config.specific.pod_spec

you get this result rendered:

---
# Source: hull-test/templates/hull.yaml
apiVersion: v1
data:
  test-get: map[initContainers:map[init:map[args:[or use this] image:map[repository:extreg
    tag:youngest] livenessProbe:map[failureThreshold:23 httpGet:map[path:/route port:876
    scheme:HTTP] initialDelaySeconds:21 periodSeconds:22 timeoutSeconds:24]]]]
  test-get-json: '{"initContainers":{"init":{"args":["or","use","this"],"image":{"repository":"extreg","tag":"youngest"},"livenessProbe":{"failureThreshold":23,"httpGet":{"path":"/route","port":876,"scheme":"HTTP"},"initialDelaySeconds":21,"periodSeconds":22,"timeoutSeconds":24}}}}'
  test-get-prettyjson: |-
    {
      "initContainers": {
        "init": {
          "args": [
            "or",
            "use",
            "this"
          ],
          "image": {
            "repository": "extreg",
            "tag": "youngest"
          },
          "livenessProbe": {
            "failureThreshold": 23,
            "httpGet": {
              "path": "/route",
              "port": 876,
              "scheme": "HTTP"
            },
            "initialDelaySeconds": 21,
            "periodSeconds": 22,
            "timeoutSeconds": 24
          }
        }
      }
    }
  test-get-rawjson: '{"initContainers":{"init":{"args":["or","use","this"],"image":{"repository":"extreg","tag":"youngest"},"livenessProbe":{"failureThreshold":23,"httpGet":{"path":"/route","port":876,"scheme":"HTTP"},"initialDelaySeconds":21,"periodSeconds":22,"timeoutSeconds":24}}}}'
  test-get-string: map[initContainers:map[init:map[args:[or use this] image:map[repository:extreg
    tag:youngest] livenessProbe:map[failureThreshold:23 httpGet:map[path:/route port:876
    scheme:HTTP] initialDelaySeconds:21 periodSeconds:22 timeoutSeconds:24]]]]
  test-get-yaml: |-
    initContainers:
      init:
        args:
        - or
        - use
        - this
        image:
          repository: extreg
          tag: youngest
        livenessProbe:
          failureThreshold: 23
          httpGet:
            path: /route
            port: 876
            scheme: HTTP
          initialDelaySeconds: 21
          periodSeconds: 22
          timeoutSeconds: 24
  test-include-code: map[auths:map[my-registry:map[auth:dXNlcm5hbWU6cGFzc3dvcmQ= email:email
    password:password username:username]]]
  test-include-code-json: '{"auths":{"my-registry":{"auth":"dXNlcm5hbWU6cGFzc3dvcmQ=","email":"email","password":"password","username":"username"}}}'
  test-include-code-prettyjson: |-
    {
      "auths": {
        "my-registry": {
          "auth": "dXNlcm5hbWU6cGFzc3dvcmQ=",
          "email": "email",
          "password": "password",
          "username": "username"
        }
      }
  test-include-code-rawjson: '{"auths":{"my-registry":{"auth":"dXNlcm5hbWU6cGFzc3dvcmQ=","email":"email","password":"password","username":"username"}}}'
  test-include-code-string: map[auths:map[my-registry:map[auth:dXNlcm5hbWU6cGFzc3dvcmQ=
    email:email password:password username:username]]]
  test-include-code-yaml: |-
    auths:
      my-registry:
        auth: dXNlcm5hbWU6cGFzc3dvcmQ=
        email: email
        password: password
        username: username
  test-include-flow: map[auths:map[my-registry:map[auth:dXNlcm5hbWU6cGFzc3dvcmQ= email:email
    password:password username:username]]]
kind: ConfigMap
metadata:
  annotations:
    test-get: map[initContainers:map[init:map[args:[or use this] image:map[repository:extreg
      tag:youngest] livenessProbe:map[failureThreshold:23 httpGet:map[path:/route
      port:876 scheme:HTTP] initialDelaySeconds:21 periodSeconds:22 timeoutSeconds:24]]]]
    test-get-json: '{"initContainers":{"init":{"args":["or","use","this"],"image":{"repository":"extreg","tag":"youngest"},"livenessProbe":{"failureThreshold":23,"httpGet":{"path":"/route","port":876,"scheme":"HTTP"},"initialDelaySeconds":21,"periodSeconds":22,"timeoutSeconds":24}}}}'
    test-get-prettyjson: |-
      {
        "initContainers": {
          "init": {
            "args": [
              "or",
              "use",
              "this"
            ],
            "image": {
              "repository": "extreg",
              "tag": "youngest"
            },
            "livenessProbe": {
              "failureThreshold": 23,
              "httpGet": {
                "path": "/route",
                "port": 876,
                "scheme": "HTTP"
              },
              "initialDelaySeconds": 21,
              "periodSeconds": 22,
              "timeoutSeconds": 24
            }
          }
        }
      }
    test-get-rawjson: '{"initContainers":{"init":{"args":["or","use","this"],"image":{"repository":"extreg","tag":"youngest"},"livenessProbe":{"failureThreshold":23,"httpGet":{"path":"/route","port":876,"scheme":"HTTP"},"initialDelaySeconds":21,"periodSeconds":22,"timeoutSeconds":24}}}}'
    test-get-string: map[initContainers:map[init:map[args:[or use this] image:map[repository:extreg
      tag:youngest] livenessProbe:map[failureThreshold:23 httpGet:map[path:/route
      port:876 scheme:HTTP] initialDelaySeconds:21 periodSeconds:22 timeoutSeconds:24]]]]
    test-get-yaml: |-
      initContainers:
        init:
          args:
          - or
          - use
          - this
          image:
            repository: extreg
            tag: youngest
          livenessProbe:
            failureThreshold: 23
            httpGet:
              path: /route
              port: 876
              scheme: HTTP
            initialDelaySeconds: 21
            periodSeconds: 22
            timeoutSeconds: 24
    test-include-code: map[auths:map[my-registry:map[auth:dXNlcm5hbWU6cGFzc3dvcmQ=
      email:email password:password username:username]]]
    test-include-code-json: '{"auths":{"my-registry":{"auth":"dXNlcm5hbWU6cGFzc3dvcmQ=","email":"email","password":"password","username":"username"}}}'
    test-include-code-prettyjson: |-
      {
        "auths": {
          "my-registry": {
            "auth": "dXNlcm5hbWU6cGFzc3dvcmQ=",
            "email": "email",
            "password": "password",
            "username": "username"
          }
        }
      }
    test-include-code-rawjson: '{"auths":{"my-registry":{"auth":"dXNlcm5hbWU6cGFzc3dvcmQ=","email":"email","password":"password","username":"username"}}}'
    test-include-code-string: map[auths:map[my-registry:map[auth:dXNlcm5hbWU6cGFzc3dvcmQ=
      email:email password:password username:username]]]
    test-include-code-yaml: |-
      auths:
        my-registry:
          auth: dXNlcm5hbWU6cGFzc3dvcmQ=
          email: email
          password: password
          username: username
  name: release-name-hull-test-test-serializing
  namespace: a-namespace-to-render

This will give some nice options for populating labels and annotations with serialized content. And it is a usable short way of populating the inline content of ConfigMaps/Secrets with string data serialized as needed.

It also applies to containers env values which I forgot in my previous list of where values are converted to string.

Thank you very much, @gre9ory ! I'm looking forward for fetching a new release and putting to use the new feature.

Am trying to wrap this up and then create some new releases containing the changeable secret type as well.

Hope to finish that this week.

Apologize for the delay, one thing lead to another and more and more things needed to be handled to wrap things up.

But I am positive it will bring a nice feature boost when it is done. I expect releases maybe end of this week, latest next week.

Great news, thank you for doing a great job!

Ok now this is done - turned out to be a really extensive look into what is going on currently and what should be going on ideally. I hope now it is a usable and consistent solution with the changes made.

First I rewrote ConfigMap and Secret implementation to use the same code (besides the base64 encode of secret data). There should come no unexpected differences in behavior when using either from the usage side. Then I made sure that handling of inline and what is in a path file is also treated the same - given that both is in string form.

For inline I expanded the scope of use so that you can now directly write your content in dictionary or list form instead of only strings. Dictionaries and lists can be serialized easily either implicitly by file extension or explicitly by a serialization instruction. You can also get or reference a dictionary or list via a _HT* or _HT/ and have it serialized as you need. I recommed to check out the updated secrets and configmaps documentation.

Regarding the original problem mentioned here you now have more ways of solving it:

hull:
  objects:
    secret:
      my-secret-1:
        data:
          .dockerconfigjson:
            inline: _HT/toJson|myUtilities.generateDotDockerConfigJson

hull:
  objects:
    secret:
      my-secret-1:
        data:
          .dockerconfigjson:
            inline: _HT/myUtilities.generateDotDockerConfigJson
            serialization: toJson

Both convert your dictionary data to a JSON string, the first notation one you can also use with annotations or env fields where you may want to store JSON data.

Let me know if that closes this whole issue from your side.

Releases:

1.29.2 1.28.8 1.27.10

Thank you very much, @gre9ory!

I like the idea of the serialization parameter, will utilize the new feature in my charts.

@khmarochos Also closing this one here since all that came out of it is implemented. The outcome very benefitial to the overall project so again much obliged for the input!

Please reopen if some issues come up. Thanks!

vidispine / hull