search

logstash-plugins / logstash-output-kafka

Kafka Output for Logstash
Apache License 2.0
74 stars 76 forks source link

Where is files location when kafka service down? #181

Open TiaraH opened 6 years ago

TiaraH commented 6 years ago

That logstash failed to send file to kafka will retry till send success. But when logstash retrying, where are the files? Is there any path queue in logstash or just in memory? If we enable Persistent Queues the files will in the queue or not?

jordansissel commented 6 years ago

There are no files specific to the Kafka output. It will retry until successful or until the configured (infinity, by default) retry count is exceeded.

If Kafka is down, Logstash will begin congestion control processes which will stall the pipeline much like a traffic jam with cars on roads -- no cars are lost in a traffic jam, they just get stuck for a while as the transit channel is congested waiting for things to exit the highway (the pipeline, in the case of Logstash).

TiaraH commented 6 years ago

Thanks for your explanation. And another questions, when kafka failed that logstash will retry until successful, but if there kafka status was normal just get some error like:"[WARN ][org.apache.kafka.clients.NetworkClient] [Producer clientId=producer-1] Error while fetching metadata with correlation id 4984 : {XXXX=UNKNOWN_TOPIC_OR_PARTITION} (kafka truly doesn't have this topic) ", if this error message, where can I move this error file? or it will retry again and again?