search

openshift / cluster-monitoring-operator

Manage the OpenShift monitoring stack
Apache License 2.0
247 stars 356 forks source link

Support enableRemoteWriteReceiver flag in prometheus{,K8s} config #2217

Open jirislav opened 7 months ago

jirislav commented 7 months ago

I need to deliver metrics from the k6-operator (testing framework from Grafana, runs in jobs) to Prometheus.

At the time of writing this issue, K6 officially supports only the Prometheus remote write as the only way to get the metrics there.

Since the CMO uses CoreOS Prometheus operator under the hood, it should be possible to extend the configuration in such a way so that the following is possible:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
spec:
  enableRemoteWriteReceiver: true

By default, Prometheus has this feature disabled, resulting in 404 Not Found when trying to use the API:

{"error":"got status code: 404 instead expected a 2xx successful status code","level":"error","msg":"Failed to send the time series data to the endpoint","output":"Prometheus remote write"}

I would expect this to be configured in the Cluster Monitoring Operator configuration in a way similar to these drafts:

kind: ConfigMap
apiVersion: v1
data:
  config.yaml: |
    prometheusK8s:
      enableRemoteWriteReceiver: true
kind: ConfigMap
apiVersion: v1
data:
  config.yaml: |
    prometheus:
      enableRemoteWriteReceiver: true
jakovrebic commented 6 months ago

@jirislav Any news on this? I'm having exact same issue only trying to connect Grafana Tempo as remote write source to Prometheus

jirislav commented 6 months ago

@jirislav Any news on this? I'm having exact same issue only trying to connect Grafana Tempo as remote write source to Prometheus

No, I have no news, we ended up not using the integration, sadly.

openshift-bot commented 3 months ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

jirislav commented 3 months ago

/remove-lifecycle stale

openshift-bot commented 6 days ago

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

jirislav commented 6 days ago

/remove-lifecycle stale