Helm chart: Allow referencing secrets for s3 storage configuration

[tchinmai7]

Is your feature request related to a problem? Please describe. The storage configuration for loki's helm chart currently requires users to pass in the secretKey and accessKey as plain-text in the values. This makes gitops driven installations difficult, as secrets must either be injected by a plugin OR stored in values (bad)

Describe the solution you'd like Allow passing in secretRef's for the accessKey and SecretKey

[amannagpal2]

how you configured accesskey then?

[sebhoss]

We use kustomize exec functions to modify the configmap and inject secrets like this:

1. Create kustomization.yaml like this:

   apiVersion: kustomize.config.k8s.io/v1beta1
   kind: Kustomization
   resources:
   - ...
   transformers:
   - transform-configmap-loki.yaml # the important part here

2. Create the transformer like this:

   apiVersion: example.com/v1
   kind: Transformer
   metadata:
   name: loki
   namespace: loki
   annotations:
   config.kubernetes.io/function: |
     exec:
       path: kustomize-exec-functions/configmap-loki.sh
   spec:
   secret-path: cluster.example/minio/tenants/loki # path in our secret store that contains credentials

3. Write the exec function:

   
   #!/usr/bin/env sh

resourceList=$(cat) # read the kind: ResourceList from stdin

kind='ConfigMap' name=$(echo "${resourceList}" | yq '.functionConfig.metadata.name' -) namespace=$(echo "${resourceList}" | yq '.functionConfig.metadata.namespace' -) secret_path=$(echo "${resourceList}" | yq '.functionConfig.spec.secret-path' -)

access_key=$(read-secret-from-store "${secret_path}" access-key) secret_key=$(read-secret-from-store "${secret_path}" secret-key)

echo "${resourceList}" | yq " ( .items[] | select(.kind == \"${kind}\" and .metadata.name == \"${name}\" and .metadata.namespace == \"${namespace}\") | .data[\"config.yaml\"]) |= (from_yaml | .common.storage.s3.access_key_id = \"${access_key}\" | .common.storage.s3.secret_access_key = \"${secret_key}\" | .ruler.storage.s3.access_key_id = \"${access_key}\" | .ruler.storage.s3.secret_access_key = \"${secret_key}\" | to_yaml)"

Kustomize will send all existing resources as a `ResourceList` into this function and we modify the ConfigMap named 'loki' on the fly and output the result.

In the future, kustomize might get the ability to modify those nested structures values directly once https://github.com/kubernetes-sigs/kustomize/issues/4517 lands.