issue : How Prometheus uses Prometheus-Kafka-Adapter to ensure that only one Prometheus delivers data to Kafka at a time.

[chenmudu]

like this : prometheus A --remote write-->prometheus-kafka-adapter|

prometheus B --remote write-->prometheus-kafka-adapter| (Only one adapter will be selected in the end)-----> Cousumer(My own App).

prometheus C --remote write-->prometheus-kafka-adapter|

That is, prometheus-kafka-adapter can select the main write remote operation.

Thanks a lot, maybe i need some help.

others: click here.

After @palmerabollo mentioned it, I updated my description as:

Three sets of P8s replicas that collect the same data source send data to their respective Adapters.

[palmerabollo]

Hi @chenmudu , I'm not sure I understand your question. You have three different prometheus sending data to the same prometheus-kafka-adapter, and you want to consume data written in kafka with your own app. Could you please clarify what you mean by "ensure that only one prometheus delivers data to kafka at a time"? Thanks.

[chenmudu]

Hi @chenmudu , I'm not sure I understand your question. You have three different prometheus sending data to the same prometheus-kafka-adapter, and you want to consume data written in kafka with your own app. Could you please clarify what you mean by "ensure that only one prometheus delivers data to kafka at a time"? Thanks.

Thanks for your reply. @palmerabollo I'm sorry I didn't describe it clearly.

Three sets of P8s replicas that collect the same data source send data to their respective Adapters.What I would like to implement is that these three P8s replica sets can implement distributed election operations, but that doesn't seem to be possible.Therefore, we want to bind to the current P8s node in the Adapter layer (weak binding), and implement a similar primary operation in the Adapter layer. Ensure that only one Adapter can send data to the Remote Endpoint (my application) at any given time.

But it seems to be strongly tied to the business codes, so I'm going to build my own adapter.

If you have a better suggestion, it seems that we can discuss it here.

Thanks a lot.

[jpfe-tid]

these three P8s replica sets can implement distributed election operations

If I understand correctly, you want to perform a leader election across all prometheus-kafka-adapters, so only the leader can send data to Kafka. It is correct?

I consider this pattern wrong because either:

1) Every prometheus instance will keep sending data to each prometheus-kafka-adapter, receiving a successful "ghost write" because it is probable that the prometheus-kafka-adapter is not a leader, therefore doesn't send data into kafka.

2) Every prometheus instance will keep sending data to each prometheus-kafka-adapter, receiving a failed write due to the prometheus-kafka-adapter not being a leader, and therefore not able/allowed to send data into kafka. These failures will increase backpressure and memory consumption on all prometheus instances that point to a non leader prometheus-kafka-adapter.

---

My suggestion is to use a different set of external labels on each prometheus instance, send all the data to kafka and filter it (unhealthy instances, etc) with your application.

https://github.com/Telefonica/prometheus-kafka-adapter/pull/64

[chenmudu]

these three P8s replica sets can implement distributed election operations

If I understand correctly, you want to perform a leader election across all prometheus-kafka-adapters, so only the leader can send data to Kafka. It is correct?

Yeah. I'm glad my description got cleaner.Thanks for your reply.

receiving a successful "ghost write"

I'm not sure what "ghost Write" is, can I understand it as an HTTP response from the prometheus-Kafka-Adapter to its sender(prometheus)?

These failures will increase backpressure and memory consumption on all prometheus instances that point to a non leader prometheus-kafka-adapter.

Prometheus writes data out of the memory to a certain extent, but I do not quite understand the meaning of "backpressure" ? Does it mean wasting some of Prometheus' own performance?Or is it something else?

My suggestion is to use a different set of external labels on each prometheus instance, send all the data to kafka and filter it (unhealthy instances, etc) with your application.

Thanks for your suggestion. I will consider it carefully after the discussion with you. ♥

64

Reading~