[collector] How to use external secrets to for secret mounts.

[indreek]

Hello

Looking for guidance. Trying to keep my git clean and secure. And i would like to use secrets for otlp receivers.

What would be the best way to include external secrets (example from Hashicorp vault) into open-telemetry collector secretMounts? https://github.com/open-telemetry/opentelemetry-helm-charts/pull/765

As far as i know there are two options: 1) Separate helm for secrets - Using tools like ArgoCD items will be in different projects. No single overview. 2) Wrapper helm chart - downside is that i need to monitor parent chart for updates and update my own chart for this.

Thanks, Indrek

[povilasv]

AFAIK: You can set environment variables for collector https://github.com/open-telemetry/opentelemetry-helm-charts/blob/main/charts/opentelemetry-collector/values.yaml#L212

And make environment variable to reflect your k8s secret. Example: https://kubernetes.io/docs/tasks/inject-data-application/distribute-credentials-secure/#define-container-environment-variables-using-secret-data

or mount secrets into pod https://github.com/open-telemetry/opentelemetry-helm-charts/blob/main/charts/opentelemetry-collector/templates/_pod.tpl#L95C25-L101

Your tool provisions the secrets, you mount them into collector and use them?

[indreek]

Your tool provisions the secrets .... what tool do you mean? Getting secrets into collector is not an issue. "Creating the secret itself" is the issue.

[povilasv]

So how do you setup secrets currently? What tool creates/manages those secrets?

[indreek]

I added option to chart to create extraObjects. https://github.com/open-telemetry/opentelemetry-helm-charts/pull/765/commits/7e6f57de09abc3c8b9d47a90002c1d6d802b7c57

Because I use ArgoCD and in order to have overview, they must be in same Helm chart. Other option would have been to make wrapper chart, but it adds regular chart maintenance task on every chart release.

[TylerHelmuth]

We could potentially have the chart create a secret.yaml, there are scenarios in the promethues charts that allow this. In general tho I am in favor of the secrets being managed separately from the helm chart; they can be installed and managed independently from the services that rely on them.

[povilasv]

You mean like this https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml#L2529-L2536 ?

This is their values.yaml code snippet that we could incorporate in collector chart:

  extraSecret:
    ## if not set, name will be auto generated
    # name: ""
    annotations: {}
    data: {}
  #   auth: |
  #     foo:$apr1$OFG3Xybp$ckL0FHDAkoXYIlH9.cysT0
  #     someoneelse:$apr1$DMZX2Z4q$6SbQIfyuLQd.xmo/P0m2c.

It looks like it allows users to provide an additional secret object https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/templates/prometheus/extrasecret.yaml#L1-L20

Would this solve the original issue?

[indreek]

Yes, I could work with something like this. It would need some modification, but in general yes. I use external-secrets. Would something like this be accepted?

[TylerHelmuth]

@povilasv ya we could allow creation a Secret object. But what is not clear to me, and why I prefer secrets be managed independently, is how you safely tell the chart the value of the secret. This kind of content:

#   auth: |
  #     foo:$apr1$OFG3Xybp$ckL0FHDAkoXYIlH9.cysT0
  #     someoneelse:$apr1$DMZX2Z4q$6SbQIfyuLQd.xmo/P0m2c.

does not seem like something you should actually be putting in a values.yaml.

[indreek]

@TylerHelmuth Like this. I can even make my git public. You still wouldn't get anything out of it.

With help of https://external-secrets.io/ i can define in my helm chart values:

extraObjects:
  - apiVersion: external-secrets.io/v1beta1
    kind: ExternalSecret
    metadata:
      name: opentelemetry-collector-tls
    spec:
      target:
        creationPolicy: 'Owner'
      data:
        - secretKey: tls.pem
          remoteRef:
            key: secrets/tools/opentelemetry
            property: opentelemetry-collector-tls
      refreshInterval: 1h
      secretStoreRef:
        kind: ClusterSecretStore
        name: vault-backend

[povilasv]

so extraSecret: wouldn't work for you, since you want to add ExternalSecret, not K8s Secret object. Based on https://github.com/open-telemetry/opentelemetry-helm-charts/pull/765, ATM, we are not willing to add extraObjects to helm charts, so you need to do it outside of the chart or use it as subchart.

Though question, do we want to add support for ExternalSecret or SealedSecret (for https://github.com/bitnami-labs/sealed-secrets) natively?

[TylerHelmuth]

We have a specific rule for templates that depend on external CRDs and right now it is only Prometheus since OTel has promises to keep with Prometheus compatibility. In general it is best if the chart includes everything it needs it order to install.

[povilasv]

Okay, so just to summarize, what we suggest in this particular case:

• use Opentelemetry Collector as a subchart or
• create ExternalSecrets / Secrets outside of the chart

[mirzaprangon]

I will add notes for reference.

The openapi collector have values as follows:

extraEnvsFrom: []
extraVolumes: []
extraVolumeMounts: []

Therefore, if you want to use external secret, for example using a secret csi driver, you will just add add the options in these keys. The secretclass for csi will have to be created separately.

This is a very common use case. I will request the team to add a part in the documentation, specially on adding endpoint and Authorization configurations using secrets.

[michaelbaltic-insight]

This is a very common use case. I will request the team to add a part in the documentation, specially on adding endpoint and Authorization configurations using secrets.

was this updated? I have secrets mounted like below, but it isnt clear how to reference them in tls:

splunk_hec:

Splunk HTTP Event Collector token.

  token: "<splunk token>"
  # URL to a Splunk instance to send data to.
  endpoint: "<splunk api>"
  # Optional Splunk source: https://docs.splunk.com/Splexicon:Source
  source: "otel"
  # Optional Splunk source type: https://docs.splunk.com/Splexicon:Sourcetype
  sourcetype: "otel"
  # Splunk index, optional name of the Splunk index targeted.
  index: "metrics"
  # Whether to disable gzip compression over HTTP. Defaults to false.
  disable_compression: false
  # HTTP timeout when sending data. Defaults to 10s.
  timeout: 10s
  tls:
    insecure_skip_verify: false
    cert_file: "what to put here"
    key_file: "what to put here"

-- Additional volumeMounts to the controller main container.

extraVolumeMounts:

• name: kvsecrets mountPath: "/mnt/kvsecrets" readOnly: true

-- Additional volumes to the controller pod.

extraVolumes:

• name: kvsecrets csi: driver: secrets-store.csi.k8s.io readOnly: true volumeAttributes: secretProviderClass: "csisecretproviderclass-otel"

  -- Additional environment variables to set

  extraEnvs:

• name: TLS valueFrom: secretKeyRef: key: tls.key name: kvinternalservicescerts