search

vectordotdev / helm-charts

Helm charts for Vector.
https://vector.dev
Mozilla Public License 2.0
113 stars 90 forks source link

Custom config using namespacing currently not doable ? #397

Closed BastienM closed 4 months ago

BastienM commented 4 months ago

Hi 👋

We are trying to break away the config from the values file (and implement config namespacing), because having a file with 2k LOC (we are going to have a lot of transforms) seems rather far away from being ideal and also prevents us from validating and testing the config ahead of release.

Alas, existingConfigMaps seems to conflicts with the use of extraVolumes and extraVolumeMounts in the StatefulSet specs.

First, it is required to escape the automagically built config, then the values provided in extraVolumeMounts seems either ignored or overridden by the current logic in charts/vector/templates/_pod.tpl: https://github.com/vectordotdev/helm-charts/blob/72ffb20576cf4141b6881ada2660fdaadf2e22ad/charts/vector/templates/_pod.tpl#L186-L192

Are we missing something to make the setup work, or does the chart currently not handle this scenario ?

Version

0.33.0

File layout

⇒ tree config
config
├── sinks
│   ├── datadog_logs.yaml
│   ├── datadog_metrics.yaml
│   ├── datadog_traces.yaml
│   └── stdout.yaml
├── sources
│   ├── datadog_agents.yaml
│   ├── vector_logs.yaml
│   └── vector_metrics.yaml
├── tests
├── transforms
│   ├── datadog_logs_vector_as_sender.yaml
│   ├── datadog_traces_vector_as_sender.yaml
│   ├── vector_logs_clean.yaml
│   ├── vector_metrics_limit_cardinalities.yaml
│   └── vector_metrics_router.yaml
└── vector.yaml

Validating the config

⇒ vector validate -C config
√ Loaded ["config"]

Our configmap

# only printing out the keys to keep things tidy and simple
⇒ kc get configmap/config-files -n vector -o yaml | yq '.data | keys'
- sinks-datadog-logs-yaml
- sinks-datadog-metrics-yaml
- sinks-datadog-traces-yaml
- sinks-stdout-yaml
- sources-datadog-agents-yaml
- sources-vector-logs-yaml
- sources-vector-metrics-yaml
- transforms-datadog-logs-vector-as-sender-yaml
- transforms-datadog-traces-vector-as-sender-yaml
- transforms-vector-logs-clean-yaml
- transforms-vector-metrics-limit-cardinalities-yaml
- transforms-vector-metrics-router-yaml
- vector-yaml

Values file

# disclaimer: this is a simplified version, from the one we are using, for clarity and readibility
role: Aggregator

existingConfigMaps:
  - config-files

extraVolumes:
  - name: config-files-sinks-datadog-logs-yaml
    configMap:
      name: config-files
      items:
        - key: sinks-datadog-logs-yaml
          path: sinks-datadog-logs-yaml
  - name: config-files-sinks-datadog-metrics-yaml
    configMap:
      name: config-files
      items:
        - key: sinks-datadog-metrics-yaml
          path: sinks-datadog-metrics-yaml
  - name: config-files-sinks-datadog-traces-yaml
    configMap:
      name: config-files
      items:
        - key: sinks-datadog-traces-yaml
          path: sinks-datadog-traces-yaml
  - name: config-files-sinks-stdout-yaml
    configMap:
      name: config-files
      items:
        - key: sinks-stdout-yaml
          path: sinks-stdout-yaml
  - name: config-files-sources-datadog-agents-yaml
    configMap:
      name: config-files
      items:
        - key: sources-datadog-agents-yaml
          path: sources-datadog-agents-yaml
  - name: config-files-sources-vector-logs-yaml
    configMap:
      name: config-files
      items:
        - key: sources-vector-logs-yaml
          path: sources-vector-logs-yaml
  - name: config-files-sources-vector-metrics-yaml
    configMap:
      name: config-files
      items:
        - key: sources-vector-metrics-yaml
          path: sources-vector-metrics-yaml
  - name: config-files-transforms-datadog-logs-vector-as-sender-yaml
    configMap:
      name: config-files
      items:
        - key: transforms-datadog-logs-vector-as-sender-yaml
          path: transforms-datadog-logs-vector-as-sender-yaml
  - name: config-files-transforms-datadog-traces-vector-as-sender-yaml
    configMap:
      name: config-files
      items:
        - key: transforms-datadog-traces-vector-as-sender-yaml
          path: transforms-datadog-traces-vector-as-sender-yaml
  - name: config-files-transforms-vector-logs-clean-yaml
    configMap:
      name: config-files
      items:
        - key: transforms-vector-logs-clean-yaml
          path: transforms-vector-logs-clean-yaml
  - name: config-files-transforms-vector-metrics-limit-cardinalities-yaml
    configMap:
      name: config-files
      items:
        - key: transforms-vector-metrics-limit-cardinalities-yaml
          path: transforms-vector-metrics-limit-cardinalities-yaml
  - name: config-files-transforms-vector-metrics-router-yaml
    configMap:
      name: config-files
      items:
        - key: transforms-vector-metrics-router-yaml
          path: transforms-vector-metrics-router-yaml
  - name: config-files-vector-yaml
    configMap:
      name: config-files
      items:
        - key: vector-yaml
          path: vector-yaml

extraVolumesMounts:
  - name: config-files-sinks-datadog-logs-yaml
    mountPath: "/etc/vector/sinks/datadog_logs.yaml"
    readOnly: true
  - name: config-files-sinks-datadog-metrics-yaml
    mountPath: "/etc/vector/sinks/datadog_metrics.yaml"
    readOnly: true
  - name: config-files-sinks-datadog-traces-yaml
    mountPath: "/etc/vector/sinks/datadog_traces.yaml"
    readOnly: true
  - name: config-files-sinks-stdout-yaml
    mountPath: "/etc/vector/sinks/stdout.yaml"
    readOnly: true
  - name: config-files-sources-datadog-agents-yaml
    mountPath: "/etc/vector/sources/datadog_agents.yaml"
    readOnly: true
  - name: config-files-sources-vector-logs-yaml
    mountPath: "/etc/vector/sources/vector_logs.yaml"
    readOnly: true
  - name: config-files-sources-vector-metrics-yaml
    mountPath: "/etc/vector/sources/vector_metrics.yaml"
    readOnly: true
  - name: config-files-transforms-datadog-logs-vector-as-sender-yaml
    mountPath: "/etc/vector/transforms/datadog_logs_vector_as_sender.yaml"
    readOnly: true
  - name: config-files-transforms-datadog-traces-vector-as-sender-yaml
    mountPath: "/etc/vector/transforms/datadog_traces_vector_as_sender.yaml"
    readOnly: true
  - name: config-files-transforms-vector-logs-clean-yaml
    mountPath: "/etc/vector/transforms/vector_logs_clean.yaml"
    readOnly: true
  - name: config-files-transforms-vector-metrics-limit-cardinalities-yaml
    mountPath: "/etc/vector/transforms/vector_metrics_limit_cardinalities.yaml"
    readOnly: true
  - name: config-files-transforms-vector-metrics-router-yaml
    mountPath: "/etc/vector/transforms/vector_metrics_router.yaml"
    readOnly: true
  - name: config-files-vector-yaml
    mountPath: "/etc/vector/vector.yaml"
    readOnly: true

dataDir: /opt/data
persistence:
  enabled: true
  size: 1.5Gi

Generated StatefulSet resource

$ gh repo clone vectordotdev/helm-charts -- --branch vector-0.33.0 && cd helm-charts
$ cd charts/vector && helm build .
$ helm template -s templates/statefulset.yaml . --values /tmp/my-values.yaml | yq '.spec.template.spec | (.containers[].volumeMounts, .volumes)'

# .spec.template.spec.containers[].volumeMounts
- name: data
  mountPath: "/opt/data"
- name: config
  mountPath: "/etc/vector/"
  readOnly: true
- name: config
  projected:
    sources:
      - configMap:
          name: config-files

#  .spec.template.spec.volumes
- configMap:
    items:
      - key: sinks-datadog-logs-yaml
        path: sinks-datadog-logs-yaml
    name: config-files
  name: config-files-sinks-datadog-logs-yaml
- configMap:
    items:
      - key: sinks-datadog-metrics-yaml
        path: sinks-datadog-metrics-yaml
    name: config-files
  name: config-files-sinks-datadog-metrics-yaml
- configMap:
    items:
      - key: sinks-datadog-traces-yaml
        path: sinks-datadog-traces-yaml
    name: config-files
  name: config-files-sinks-datadog-traces-yaml
- configMap:
    items:
      - key: sinks-stdout-yaml
        path: sinks-stdout-yaml
    name: config-files
  name: config-files-sinks-stdout-yaml
- configMap:
    items:
      - key: sources-datadog-agents-yaml
        path: sources-datadog-agents-yaml
    name: config-files
  name: config-files-sources-datadog-agents-yaml
- configMap:
    items:
      - key: sources-vector-logs-yaml
        path: sources-vector-logs-yaml
    name: config-files
  name: config-files-sources-vector-logs-yaml
- configMap:
    items:
      - key: sources-vector-metrics-yaml
        path: sources-vector-metrics-yaml
    name: config-files
  name: config-files-sources-vector-metrics-yaml
- configMap:
    items:
      - key: transforms-datadog-logs-vector-as-sender-yaml
        path: transforms-datadog-logs-vector-as-sender-yaml
    name: config-files
  name: config-files-transforms-datadog-logs-vector-as-sender-yaml
- configMap:
    items:
      - key: transforms-datadog-traces-vector-as-sender-yaml
        path: transforms-datadog-traces-vector-as-sender-yaml
    name: config-files
  name: config-files-transforms-datadog-traces-vector-as-sender-yaml
- configMap:
    items:
      - key: transforms-vector-logs-clean-yaml
        path: transforms-vector-logs-clean-yaml
    name: config-files
  name: config-files-transforms-vector-logs-clean-yaml
- configMap:
    items:
      - key: transforms-vector-metrics-limit-cardinalities-yaml
        path: transforms-vector-metrics-limit-cardinalities-yaml
    name: config-files
  name: config-files-transforms-vector-metrics-limit-cardinalities-yaml
- configMap:
    items:
      - key: transforms-vector-metrics-router-yaml
        path: transforms-vector-metrics-router-yaml
    name: config-files
  name: config-files-transforms-vector-metrics-router-yaml
- configMap:
    items:
      - key: vector-yaml
        path: vector-yaml
    name: config-files
  name: config-files-vector-yaml
BastienM commented 4 months ago

Never mind, my fatigued brain did not noticed the typo on extraVolume(s)Mounts. Apologies for the noise.