search

deepflowio / deepflow

eBPF Observability - Distributed Tracing and Profiling
https://deepflow.io
Apache License 2.0
2.93k stars 330 forks source link

[FR] FR-23: Support prometheus for data master selection when connected to deepflow to reduce storage and network overhead #2407

Open linyongluis opened 1 year ago

linyongluis commented 1 year ago

Search before asking

Description

There are many ways to deploy prometheus clusters, most of which use replica deployment.

When writing to deepflow or other storage terminals remotely, the same data may be written multiple times.

At present, there is no better solution for the open source ecosystem, so we hope to solve this troublesome thing here.In order to reduce the storage and network pressure of deepflow.

Use case

prometheus high availability

Related issues

No response

Are you willing to submit a PR?

Code of Conduct

chenmudu commented 1 year ago

@linyongluis Hi, here are some of the previous discussions for reference, also some articles for reference.

There are some points worth discussing, such as the fact that tagging does not reduce data transfer and network consumption. The same time-series data is still sent to a third party such as deepflow.

If someone else has a better discussion, it can happen here.

linyongluis commented 1 year ago

Chen Chen and I co-designed the Prometheus adapter, you can view the documentation: https://docs.qq.com/doc/DZW11UkJaSGFSRlpt?&u=95d9aeed1f614408bd954b503a570d7d

chenmudu commented 1 year ago

@linyongluis could not continue to participate in this work due to some personal circumstances. @taloric and @chenmudu will finish the work next.