[question] Is it possible to create ConfigMap from a directory using glob pattern ?

[ievgenii-shepeliuk]

Hello

In pure Helm it's possible to declare template like this, that will create ConfigMap with an entry for every yaml file in mydir folder.

apiVersion: v1
kind: ConfigMap
metadata:
  name: mydir
data:
  {{- (.Files.Glob "mydir/*.yaml").AsConfig | nindent 2 }}

Is it possible to achieve the same with HULL ? I tried to use _HT!{{ ... }} but without success.

[gre9ory]

Hi @ievgenii-shepeliuk,

yes that is possible. You can use a _HT! transformation on the data property similar to your Helm example.

Some aspects to consider to make it work:

• since you are using a _HT! transformation, you need to use (index . "$") instead of just . or $
• the structure of the data to create needs to be HULL's data structure as documented here and not the Kubernetes data structure which consists only of (file) keys and (file) contents. HULL first executes all transformations and then processes the resulting structures, therefore the result of the transformation here needs to be a HULL data structure.
• the AsConfig function from your example cannot be used since under the hood this Helm function creates key-value pairs from the files and their read contents. To create HULL's more expressive data strructure, you can range over the files and create the individual HULL data entries. For each individual data entry you can simply set the path property to have HULL read the file contents for you from the path.

---

Assuming in a subdir mydir you have two files:

• test_1.yaml with content name: i am test_1.yaml
• test_2.yaml with content name: i am test_2.yaml

this way it should work to load them into a ConfigMap:

hull:
  objects:
    configmap:
      from-folder:
        data: |-
          _HT!
            {
              {{ range $path, $_ := (index . "$").Files.Glob "mydir/*.yaml" }}
              {{ (base $path) }}: { 'path': {{ $path }} },
              {{ end }}
            }

The generated ConfigMap should look like this:

apiVersion: v1
data:
  test_1.yaml: 'name: i am test_1.yaml'
  test_2.yaml: 'name: i am test_2.yaml'
kind: ConfigMap
metadata:
  ...

Hope this works for you, please let me know if not. Thank You!

[eshepelyuk]

Hello @gre9ory

The good thing about this answer is that it worked.

But the bad thing is - that I have no idea why and how it's implemented. In particular this block

{{ (base $path) }}: { 'path': {{ $path }} },

I've been trying to change the code to see the changes in the outcome, but only this particular code worked. It's not a standard Helm template, but what is it then ? Why does it work ? What is the meaning of {'path' ..} construct, what is the trailing comma for ?

[gre9ory]

I'll try to explain.

Let us write the dynamically generated ConfigMap from files example without the Files.Glob part. So this is how you would statically add the files one by one to a ConfigMap in HULL:

hull:
  objects:
    configmap:
      from-folder:
        data:
          test_1.yaml: 
            path: mydir/test_1.yaml
          test_2.yaml: 
            path: mydir/test_2.yaml

The above code piece:

        data:
          test_1.yaml: 
            path: mydir/test_1.yaml
          test_2.yaml: 
            path: mydir/test_2.yaml

is the HULL syntax for specifying data entries. You can load the content from a path or use the inline field to specify content directly here. HULL aims to provide comfort features here to simplify adding ConfigMap data to your deployment. It is different from the rendered Kubernetes ConfigMap output which just has key-value pairs as data. This above structure is what we need to generate by dynamically creating the entries from the found Files.Glob pattern.

When dynamically creating larger dictionary structures via _HT!, it is advisable to use the flow-style YAML syntax because you won't get into any trrouble because of messed up indentation. In flow-style, the static data block from above can look like this:

data:
  {
    'test_1.yaml': { 'path': 'mydir/test_1.yaml' },
    'test_2.yaml': { 'path': 'mydir/test_2.yaml' }
  }

The key about flow-style is that indentation is irrelevant. You could also write it more compact like this:

data: {'test_1.yaml': {'path': 'mydir/test_1.yaml'}, 'test_2.yaml': {'path': 'mydir/test_2.yaml'}}

It is notation-wise very close to JSON as you see. Like in JSON you need to remember to quote strings. But all-in-all it is the safest way to create complex data structures as a result of _HT! transformations.

---

In summary, the snippet dynamically creates a YAML flow-style data structure which is compatible with HULL from the Files.Glob result. It does so by iterating over the found files and creates HULL-conforming data entries in YAML flow-style. The , is needed in flow-style YAML to seperate dictionary entries (thankfully the last trailing , is ignored by the YAML parsing engine so no special logic is needed to remove it). The data entries are then processed by HULL and create the Kubernetes compliant

data:
  test_1.yaml: 'name: i am test_1.yaml'
  test_2.yaml: 'name: i am test_2.yaml'

result.

---

As a side note, in this case you could also have used the plain block-style YAML which worked too in my test:

data: |-
          _HT!
            {{ range $path, $_ := (index . "$").Files.Glob "files/mydir/*.yaml" }}
              {{ base $path }}:
                path: {{ $path }}
            {{ end }}

but I really don't recommend block-style here it because it is prone to fail due the intricacies of indentation.

Hope this explains it well enough, otherwise I am happy to provide more details.

[eshepelyuk]

First of all, thank you for providing the solution and detailled explanation.

2nd, in my experience such Helm pattern (ConfigMap from a folder) is a quite common. Do you think it could be introduced as additional functionality for configmap objects ?

Something like this

hull:
  objects:
    configmap:
      mymap:
        data:
          globAsConfig: mydir/*.yaml

[gre9ory]

A good idea.

Easiest to provide an include function like hull.util.virtualdata.glob that can generate entries from a Glob pattern. The same function would work for Secrets and ConfigMaps since HULL treats them equally on the input side.

You could use it like this to generate the entries from Glob:

hull:
  objects:
    configmap:
      mymap:
        data: _HT/hull.util.virtualdata.glob:GLOB:"mydir/*.yaml"

Would that work?

[eshepelyuk]

A good idea.

Easiest to provide an include function like hull.util.virtualdata.glob that can generate entries from a Glob pattern. The same function would work for Secrets and ConfigMaps since HULL treats them equally on the input side.

You could use it like this to generate the entries from Glob:

hull:
  objects:
    configmap:
      mymap:
        data: _HT/hull.util.virtualdata.glob:GLOB:"mydir/*.yaml"

Would that work?

Yes, that would be great. I just never saw this _HT/.. notation.

[gre9ory]

No worries, it is a general shortcut to call any function in your charts templates. You can read about it here.

So by adding a define such as

{{- define "hull.util.virtualdata.glob" -}}
{{- $parent := (index . "PARENT_CONTEXT") -}}
{{- $glob := (index . "GLOB") -}}
...
{{- end -}}

to the templates you can call it this way (with arguments).

By the way you can also define and add your own custom define's and call them this way. Could be of help to again add a reusable function layer to another library chart so your HULL based helm charts can utilize them. If you are interested in this advanced usecase check out hull-vidispine-addon where we have defined some reusable functions for our companies helm charts.

[eshepelyuk]

Excuse me coming with another question @gre9ory

Is it possible with HULL to include a file content into arbitrary field of custom resource

I am having smth like this - I want to literally include a file into CRD YAML content field.

hull:
  objects:
    customresource:
      my-res:
        apiVersion: ...
        kind: ...
        spec:
          sources:
            - name: my.java
              content: |
                 ....

With regular helm I would do this with {{ .Files.Get "src/my.java" }}.

I've tried to use _HT!{{ (index . "$").Files.Get "src/my.java" }} but as usual no luck.

[gre9ory]

Sorry to hear that you usually have no luck. Your syntax looks quite alright, the problem is just a different one here. It comes down to a limitation of HULL, which is that - at least right now - it is not possible to use transformations inside array elements. You can apply transformations on dictionary values and as such create a complete array in the transformation but you cannot transform just individual elements of the array.

Technically, when scanning the nested dictionary - which is the values.yaml - for transformations, any string value that represents a transformation is replaced with the transformations result. Once you enter an array structure this does not work anymore. Modifying arrays is not as easy as replacing values of a dictionary. You would need to keep track of array element order, cache the element and recursively process the whole subtree before being able to insert it back into the dictionary tree at the array elements position. Not saying it is impossible but feels like a more challenging task.

HULL is designed to avoid arrays as much as possible. Arrays in the context of Helm and merging of configuration layers are just inconvenient to work with. Arrays need a proper schema to define how they may be mergeable via a given property (like the Kubernetes schema does have annotations for) - Helm does not have a solution for that right now. Hence HULL treats some important, high-level array structures from Kubernetes as dictionaries. For the most part, this is fine. If you encounter array structures at more detailed object configuration levels you typically don't mess with them anymore configuration wise. In CustomResources you may of course encounter arrays at any level making it a little problematic in this regard.

Some related discussion points are here and here.

I understand that this aspect may be irritating to people coming to this project. Maybe I'll try to tackle the problem if I find the time for it, so far I got around it myself pretty fine.

Nevertheless, you may be able to solve your concrete case with a little bit of rewriting. Given that your array structure does not get (much) deeper than what you have shown here, you just need to return the whole array for sources. This will be fine for rather plain structures in the array but will probably get messy when your array elements are getting more complex-ish.

hull:
  objects:
    customresource:
      my-res:
        spec:
          sources: |- 
            _HT!
              [ 
                { 
                  'name': 'my.java', 
                  'content': '{{ (index . "$").Files.Get "src/my.java" }}'
                }
              ]

---

Just to mention it: I am convinced that you can achieve almost everything satsifactory with HULL which you could achieve with plain Helm and more - but it is not a one way street. You don't have to exclusively use it. If a plain Helm template solves your problem you can write and use it as in any other Helm chart. It will just not be integrated into the HULL logic but maybe it doesn't need to be to solve your issue alright.

By the way: we will provide the hull.util.virtualdata.glob function for direct use in a release sometime soon.

[ievgenii-shepeliuk]

Thanks for the suggestion, unfortunately it hadn't worked for me. Entire sources is rendered as null. So I am going to plain Helm templates.

[gre9ory]

Hmm. did test that snippet beforehand, It was working alright for me.

---

In subfolder src i have a file my.java with some random content like:

Imagine Java Code lines ...

Then in a test customresource i put this:

hull:
  objects:
    customresource:
      mail-receiver:
        apiVersion: "mailReceiverApi/v1"
        kind: "MailReceiver"
        spec:
          incomingAddress: "othermailaddress@mail.com"
          sources: |- 
            _HT!
              [ 
                { 
                  'name': 'my.java', 
                  'content': '{{ (index . "$").Files.Get "src/my.java" }}'
                }
              ]

It renders this object out as expected:

---
# Source: hull-test/templates/hull.yaml
apiVersion: mailReceiverApi/v1
kind: MailReceiver
metadata:
  labels:
    app.kubernetes.io/component: mail-receiver
    app.kubernetes.io/instance: release-name
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: hull-test
    app.kubernetes.io/part-of: undefined
    app.kubernetes.io/version: 1.29.0
    helm.sh/chart: hull-test-1.29.0
  name: release-name-hull-test-mail-receiver
  namespace: default
spec:
  incomingAddress: othermailaddress@mail.com
  sources:
  - content: 'Imagine Java Code lines ... '
    name: my.java

---

I think either your path(s) have a problem, there is a typo/indentation problem elsewhere in the code block or there is something inside the files loaded that unexpectedly breaks it. I just used a simple string for a quick test,

I have a couple of questions if you may:

Do you have {{ or }} templating placeholders in the file(s)? Can you share the actual contents of the files so I can test it on my end? If not of course this is fine ... Does it work as expected with the plain Helm template? Can you share this plain template too?

Your response is appreciated, thanks!

[eshepelyuk]

1. Sample lines worked for me too, the real files did not.
2. Actual Java code and actual YAML I've tried - failed.
3. Unfortunately I can't share the actual code.
4. I don't have placeholders {{ or }} in my java and yaml files.
5. My java and yaml files work fine with plain Helm templates.

   apiVersion: camel.apache.org/v1
   kind: Integration
   metadata:
     name: ingest
   spec:
     sources:
       - name: IngestDSL.java
         language: java
         content: |
           {{ .Files.Get "src/kamel/java/IngestDSL.java" | nindent 8 }}

[gre9ory]

Ok thanks for the information! Seems obvious the problem(s) lie(s) in the processing of complex file content actually.

To "reproduce" I grabbed a rather short real Java code example and also it did render to null. So same as in your case when importing it into the custom resource. The same code example did however import fine into a ConfigMap from an external file via the HULL mechanism.

Looked around a bit and found eg. this SO topic. It inspired me to try to add a toYaml to the file contents (which I found to also happen in the ConfigMap code).

Now this worked with my (simple) example Java code from the looks of it:

sources: |- 
  _HT!
    [ 
      { 
        'name': 'my.java', 
        'content': {{ (index . "$").Files.Get (printf "%s" "src/my.java") | toYaml }}
      }
    ]

yielding

sources:
  - content: "class Main {\r\n\r\n  public static void main(String[] args) {\r\n    \r\n
      \   int first = 10;\r\n    int second = 20;\r\n\r\n    \r\n    int sum = first
      + second;\r\n    System.out.println(first + \" + \" + second + \" = \"  + sum);\r\n
      \ }\r\n}"
    name: my.java

As to why you'd have explicitlytoYaml a multiline string here I have to admit I am not entirely sure.

Would be much appreciated if you could give that a try with your concrete cases to know that does work across the board?

---

By the way, I also had success with the "indented" YAML approach in my test. So if indented correctly this also does work but still recommend the flow-style for more robustness :)

sourcesIndented: |- 
  _HT!
  - name: 'my.java' 
    content: |
    {{ (index . "$").Files.Get "src/my.java" | toYaml | indent 4}}

[eshepelyuk]

Thanks once again, but it still doesn't work for me :( So let's stop here, will use plain Helm.

[gre9ory]

Sorry, would like to get to the bottom of it but without the actual files it seems to be not possible ... thanks for the valid input anyways.

[eshepelyuk]

Hello @gre9ory

Finally I've achieved smth very close to what I wanted with HULL using hybrid HULL/Helm approach for quite complex YAML file.

values.yaml

hull:
objects:
customresource:
dbzm-kamel:
apiVersion: camel.apache.org/v1
kind: Integration
spec:
sources:
- name: dbzm.yaml
language: yaml
content: |-
_HT/util.dbzm.src

templates/_utils.tpl

{{- define "util.dbzm.src" -}}
{{ (index . "PARENT_CONTEXT").Files.Get "src/main/kamel/dbzm.yaml" }}
{{- end -}}

Thanks for all previous explanations and inspirations.

[gre9ory]

Glad you found something that works for you!

One (maybe last) suggestion, you could parameterize and thus generalize the helper function like this so you can reuse it with different paths:

{{- define "util.import.path" -}}
{{- $path := (index . "PATH") -}}
{{ (index . "PARENT_CONTEXT").Files.Get $path }}
{{- end -}}

Call it with:

hull:
  objects:
    customresource:
      dbzm-kamel:
        apiVersion: camel.apache.org/v1
        kind: Integration
        spec:
          sources:
            - name: dbzm.yaml
              language: yaml
              content: |-
                _HT/util.import.path:PATH:"src/main/kamel/dbzm.yaml"

I think extending the HULL charts with shared functions is a good thing and totally valid. The intention of HULL is to be a toolkit on which you can build your own stuff in form of functions - just adding regular template files should be avoided.

We do package a lot of shared helper functions for our business logic in a seperate library chart and include them besides HULL. This way you can really boost reusability for your own use cases and can reduce repeated code.

[eshepelyuk]

Thanks, I've already did it in another project where I needed that :)))

We do package a lot of shared helper functions for our business logic in a seperate library chart and include them besides HULL. This way you can really boost reusability for your own use cases and can reduce repeated code.

Yeah, that's absolutely true but the same time it's what I'm trying to avoid by adopting HULL - i.e. creating common repo of reusable Helm templates/functions. It took too much time for maintaining in the past.

[gre9ory]

Helper functions:

hull.util.tools.virtualdata.data.glob hull.util.tools.file.get

are available in releases

1.28.15 1.29.8 1.30.1

[eshepelyuk]

Helper functions:

hull.util.tools.virtualdata.data.glob hull.util.tools.file.get

are available in releases

1.28.15 1.29.8 1.30.1

Thank you.