search

Telefonica / prometheus-kafka-adapter

Use Kafka as a remote storage database for Prometheus (remote write only)
Apache License 2.0
364 stars 135 forks source link

Batch num messages doesn't work - kafka recieve metrics one by one #58

Closed shayyashar closed 3 years ago

shayyashar commented 3 years ago

Hi, Im trying to take metrics from peometheus with adapter and batch messages of 100 but I see at kafka metrics one by one. Someone who can help? Thnx :)

shayyashar commented 3 years ago

I meant that the producer send to kafka json that represents one metrics, and not json of jsons that represents more than one metric in one message.

{ { Metric 1 }, ... { Metric 100 } }

palmerabollo commented 3 years ago

Hi @shayyashar, I guess you are using KAFKA_BATCH_NUM_MESSAGES: 100, right? That is the expected behavior. Batching them means that they are sent in batches of 100 entries, but they are not grouped together in a complex/nested structure.

I mean, they are sent as

{Metric 1},
{Metric 2},
...
{Metric 100}

and not

[
  {Metric 1},
  {Metric 2},
  ...
  {Metric 100}
]

It makes sense because consumers don't need to be aware of wether they were sent in a batch or not. I hope it helped. I'm closing this issue.