grafana / mimir

Grafana Mimir provides horizontally scalable, highly available, multi-tenant, long-term storage for Prometheus.
https://grafana.com/oss/mimir/
GNU Affero General Public License v3.0
4.1k stars 526 forks source link

GCS service account credential doesn't format correctly when using ENV var #9649

Open zakris-s opened 6 days ago

zakris-s commented 6 days ago

Describe the bug

When using an environment variable for the GCS service account key JSON data Mimir pods that depend on that credential won't start with the error code err="invalid character 't' looking for beginning of value failed to create credentials from JSON.

To Reproduce

Steps to reproduce the behavior:

  1. Specify an ENV var contained GCS service account JSON data like this:
    common:
    storage:
     backend: gcs
      gcs:
        service_account: |
          ${GCS_SA_VAR_NAME}
  2. Install Mimir via the helm chart

Expected behavior

Mimir will start normally and use the above VAR as the GCS credentials to access the bucket

Environment

Additional Context

Note: If I copy the ENV var from one of the pods and paste it in to the above config it works as expected.

narqo commented 4 days ago

This looks similar to one outlined in #8704. Could you check if changing the config to one below fixes it?

common:
  storage:
    backend: gcs
      gcs:
        service_account: >-
          ${GCS_SA_VAR_NAME}

(note the >- to fold the new lines from the property's value).

zakris-s commented 23 hours ago

Yeah that is the issue I actually came from, and that solution does not work. Using '|' or '|-' to fold the lines also does not work for me.