search

Telefonica / prometheus-kafka-adapter

Use Kafka as a remote storage database for Prometheus (remote write only)
Apache License 2.0
364 stars 135 forks source link

error: Local: Queue full, msg: couldn't produce massage in kafka #90

Open golangav opened 2 years ago

golangav commented 2 years ago

Hello

This error is also an error report when the network is not available, which is easy to mislead. Restart and return to normal

Can you distinguish it

https://github.com/confluentinc/confluent-kafka-go/issues/346

image image
palmerabollo commented 2 years ago

Hi @golangav, thanks for opening this issue. What do you recommend us to do when the queue is full?

golangav commented 2 years ago

Hello, can you add a queue

or batch.size linger.ms

golangav commented 2 years ago

@palmerabollo Hello, can you add a queue https://github.com/confluentinc/confluent-kafka-dotnet/issues/703

image

queue. buffering. Max.kbytes to specify how much information can be cached. The default is 400MB, and the maximum can be increased to 2097151kb, that is, about 2GB (because the cache is not 100% of your messages, but also has key or other meta information, so the actual available can not reach 2GB).

queue. buffering. Max.messages to specify how many messages can be cached. The default is 100000. That's enough.

Errors such as buffererror: Local: queue full can be avoided by changing these two properties and optimizing the process control of the program.

sschroed99 commented 3 months ago

Stupid question, the above 2 recommendations should be adjusted on the adapter side or Kafka cluster side?