Is remote write / Thanos receive to a different cluster a supported feature ?

[jomeier]

Hi,

I have a few questions:

• is Prometheus Remote Write to a Thanos Receiver that maybe is located in a different (!) cluster a supported feature? This would enable us to have a centralized monitoring for different OpenShift clusters. At least we hope so.
• does Remote Write send cluster metrics to a remote location ?
• does Remote Write send user defined workload metrics to a remote location ?

I saw this blog article but there is no hint if remote write is a supported feature nowadays: https://www.openshift.com/blog/federated-prometheus-with-thanos-receive

Thanks a lot and greetings,

Josef

[sichvoge]

Hi @jomeier. Very good question. As the blog post already hints we do not support a remote Thanos receiver deployment neither do we support configuring remote write at the moment. Nevertheless, our goal is to support the remote-write configuration some time in the beginning of next year. In the meanwhile, you can use it as it is already exposed via our ConfigMap like in the example below.

config.yaml: |
    prometheusK8s:
      remoteWrite:
      - url: "https://test.remotewrite.com/api/write"

Again, since this is in dev preview you may not find any documentation. Additionally, we currently have no plans to support your own Thanos+Thanos receiver deployment.

Now to your other questions, yes remote-write allows you send metrics to a remote location that exposes an API that implements that remote-write API (e.g. Thanos receiver but there are others[1]).

Hope that makes sense but let me know if you still have questions.

[1] https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage

[jomeier]

@sichvoge Thank you for your response.

It is absolutely ok if you don't support external components (Thanos recevier) because that is in the users hands.

More important is, that you plan to officially support remote write because it seems to enable us to implement a central multi cluster OpenShift monitoring. We are missing that feature since a while.

This remote write / Thanos receive combination could enable that.

In one of the last OpenShift roadmap presentations I saw a hint that a central monitoring for OpenShift is on the roadmap. Is this true and will it be implemented with remote write / Thanos receive?

[sichvoge]

There are probably multiple angles to your question. Yes, we have making remote-write generally available in our roadmap for early next year. That will help you to configure it in case you already have, or think about deploying your own, central system.

Now to the topic that Red Hat provides a central system. That is already out with the Advanced Cluster Manager and this solution is using remote-write indeed.

[sichvoge]

/label Question

[openshift-ci-robot]

@sichvoge: The label(s) /label Question cannot be applied. These labels are supported: platform/aws, platform/azure, platform/baremetal, platform/google, platform/libvirt, platform/openstack, ga, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, px-approved, docs-approved, qe-approved

In response to [this](https://github.com/openshift/cluster-monitoring-operator/issues/975#issuecomment-722431997): >/label Question Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[jomeier]

@sichvoge Thanks a lot for answering my questions.

I'll close this issue/question.