palmerabollo
commented
2 years ago @psreekrishnan sorry for the late reply. I've not touched that code in months, but here is the mapping between vars and librdkafka properties https://github.com/Telefonica/prometheus-kafka-adapter/blob/8d4d51dce302bb36605e0d8df2ce41d21c779ac2/main.go#L33=
KAFKA_BATCH_NUM_MESSAGES is mapped to "batch.num.messages", not "queue.buffering.max.ms", that is not supported yet, as far as I remember. Feel free to contribute that change if you think it's useful in some use cases.
manosec
Hi There,
Setting KAFKA_BATCH_NUM_MESSAGES to different values has no effect on the adapter. Tried by setting10K (default) up to 50K.
I was looking for a way to avoid the below error by increasing the BATCH_NUM_MESSAGES value hoping that I get more room in the adapter to accommodate more messages in the queue.
`{"error":"Local: Queue full","level":"error","msg":"couldn't produce message in kafka topic prometheus-metrics-pft","time":"2022-04-20T10:37:46Z"}
Configure the below variables other than the broker list. KAFKA_COMPRESSION=snappy KAFKA_BATCH_NUM_MESSAGES=30000
Also based on the librdkafka document, "queue.buffering.max.ms" is the delay in milliseconds to wait for messages in the producer queue. I am worried about the default value (set to 0) bypassing the KAFKA_BATCH_NUM_MESSAGES values.
queue.buffering.max.ms | P | 0 .. 900000 | 0 | Delay in milliseconds to wait for messages in the producer queue to accumulate before constructing message batches (MessageSets) to transmit to brokers. A higher value allows larger and more effective (less overhead, improved compression) batches of messages to accumulate at the expense of increased message delivery latency.*Type: integer* -- | -- | -- | -- | --`
Is KAFKA_BATCH_NUM_MESSAGES variable not the right variable to bypass the "Queue full" error?