Support for projected volumes at the Mount Trait

[lsergio]

Requirement

I wish I could mount multiple config maps at the same directory. My use case is that I wish I could keep each resource file required by the Integration in its own config file and have all the files mounted at /etc/camel/resources location, regardless of where they came from.

Problem

Keeping all files in a single config map makes editing a little bit harder, and may reach the 1MB size limit for config maps if the number of files grows.

Proposal

We could use projected volumes for that; The operator might detect config maps that are mapped to the same location and automatically use projected volumes to mount them. A configuration would exist to turn on this behavior.

Open questions

No response

[lsergio]

A scenario where this might be helpful is using multiple jsonata components in an Integration. Each one would require a template file.

[lburgazzoli]

I remember that we did try to do something similar in the past but we didn't succeed as there were some combinations not working, on top of my mind:

• knative not supporting projections
• the ${configmap|secret:name/key} property placeholder function not working anymore

Things may have changed so it may not be an issue anymore but it would be nice to have a minimal design doc to assess the impact.

[lburgazzoli]

A scenario where this might be helpful is using multiple jsonata components in an Integration. Each one would require a template file.

I see two way of addressing this issue:

• use ${configmap|secret:name/key}
• reference the template as a classpath resource

Just for my understanding, is any of the two options potentially addressing the problem ?

[lsergio]

@lburgazzoli I'm working around this with the following strategy:

• Created the config maps for each file:

apiVersion: v1
data:
  data: file1 contents
kind: ConfigMap
metadata:
  name: cm1
---
apiVersion: v1
data:
  data: file2 contents
kind: ConfigMap
metadata:
  name: cm2

And mounted them like this:

  traits:
    mount:
      resources:
      - configmap:cm1@/etc/camel/resources/file1.txt
      - configmap:cm2@/etc/camel/resources/file2.txt

Then the Integration references the files as:

{{RESOURCES_DIR}}/file1.txt/data

where RESOURCES_DIR is /etc/camel/resources

My goal was to use only {{RESOURCES_DIR}}/file1.txt, but still be able to use separate cms for each file, due to the size constraints.

In my case we're creating an IDE-like frontend where our users would input their integration code and config files, so using the classpath is not an option.

[lsergio]

According https://knative.dev/docs/serving/services/storage/, projected volumes are supported in Knative.

[lburgazzoli]

Just to make ti clear, I'm not trying to push back on the implementation but I'm really curious about why the other solutions are not working since some of them were implemented for this use case :)

So I may be wrong but any config map entry, is then reachable by classpath so, in theory in your component you should be able to just use i.e. file1.txt (if the component has been updated to support the resource loader).

[lsergio]

@lburgazzoli Now I got your point.

I tested it like this:

Config Maps:

apiVersion: v1
data:
  file1.txt: "{\"name\": firstName}"
kind: ConfigMap
metadata:
  name: cm1
---
apiVersion: v1
data:
  file2.txt: "{\"name\": lastName}"
kind: ConfigMap
metadata:
  name: cm2

And the Integration:

apiVersion: camel.apache.org/v1
kind: Integration
metadata:
  name: myintegration
spec:
  sources:
  - name: main.yaml
    language: yaml
    content: |-
      - from:
          uri: "rest:POST:/test"
          steps:
          - to:
              uri: "jsonata:/file1.txt?inputType=JsonString&outputType=JsonString"
          - to:
              uri: "log:info"
  traits:
    logging:
      level: DEBUG
    ingress:
      enabled: false
    knative-service:
      enabled: false
    mount:
      resources:
      - configmap:cm1
      - configmap:cm2

The files were still mounted at different directories:

k exec -it myintegration-74d7fdd854-cfbqw -- ls -R /etc/camel/resources

/etc/camel/resources:
cm1  cm2

/etc/camel/resources/cm1:
file1.txt

/etc/camel/resources/cm2:
file2.txt

but reading them from the classpath worked fine.

So, for components that can load classpath-based resources, this works fine. The use case would then be components that eventually do not support so.

Thanks for the support.

[lburgazzoli]

but indeed the feature is useful so I'd appreciate if we can make it 👍

[squakez]

Indeed. The actual way this could work is https://camel.apache.org/camel-k/2.2.x/configuration/runtime-resources.html#runtime-resource-path. However, adding the support for project volumes sounds promising. Feel free to have a look if you can contribute it.

[github-actions[bot]]

This issue has been automatically marked as stale due to 90 days of inactivity. It will be closed if no further activity occurs within 15 days. If you think that’s incorrect or the issue should never stale, please simply write any comment. Thanks for your contributions!