host & hostKey bug?

[edas-smith]

I have recently moved over to using this helm chart and have been deploying it via ArgoCD. I believe I've run into a bug where the host field is not being properly read unless its being pulled in from a secret. Using version 0.6.0

Below are samples of the values.yaml file

For example, this will NOT work as the url is not being built properly i.e, the host is not being picked up.

cluster:
  name: "my-cluster"
externalServices:
  prometheus:
    host: https://prometheus-prod-01-gb-south-1.grafana.net # Obscured
    basicAuth:
      usernameKey: username
      passwordKey: password
    secret:
      create: false
      name: "grafana-prometheus-secret"
  loki:
    host: https://logs-prod-001.grafana.net # Obscured
    basicAuth:
      usernameKey: username
      passwordKey: password
    secret:
      create: false
      name: "grafana-loki-secret"

However, this WILL work :

cluster:
  name: "my-cluster"
externalServices:
  prometheus:
    hostKey: host # Change
    basicAuth:
      usernameKey: username
      passwordKey: password
    secret:
      create: false
      name: "grafana-prometheus-secret"
  loki:
    hostKey: host # Change
    basicAuth:
      usernameKey: username
      passwordKey: password
    secret:
      create: false
      name: "grafana-loki-secret"

Looking at the grafana-grafana-agent-0 pod logs you see this :

• level=warn msg="Failed to send batch, retrying" component=prometheus.remote_write.metrics_service subcomponent=rw remote_name=12fd2f url=/api/prom/push err="Post \"/api/prom/push\": unsupported protocol scheme \"\""

  And

• level=warn msg="error sending batch, will retry" component=loki.write.grafana_cloud_loki component=client host="" status=-1 tenant="" error="Post \"/loki/api/v1/push\": unsupported protocol scheme \"\""

  I don't know if this is intentional, but it definitely seems like a bug. Figured I put this here also incase anyone has the same issue.

[ohdearaugustin]

I can confirm this so far., but still need to do some more testing.

[ohdearaugustin]

This configuration with no secret keys set would also not work:

cluster:
  name: "my-cluster"
externalServices:
  prometheus:
    secret:
      create: false
      name: "grafana-prometheus-secret"
  loki:
    secret:
      create: false
      name: "grafana-loki-secret"

I guess the charts default values are not working.

[hegerdes]

Was also facing the same issue. Using a secret with the host value set works though:

apiVersion: v1
kind: Secret
metadata:
  name: grafana-agent-prometheus-cred
  namespace: monitoring
type: Opaque
data:
  host: xxx
  username: xxx
  password: xxx

So either plain text in the helm value file for credentials, which is not a good idea. Or all values, including host in the the secret. A mix of both does not work, even if the values file suggest that it would work.

[CMarcher-Ahuora]

I can confirm this is still a problem. I was required to embed a hostKey value in each of the needed secrets.

[zktaiga]

Same here.