search

oleksiyk / kafka

Apache Kafka 0.9 client for Node
MIT License
297 stars 85 forks source link

Report topic and partition in producer errors #126

Closed koresar closed 7 years ago

koresar commented 8 years ago

Hello.

Was trying to attach topic and partition to the producer errors. Couldn't implement that successfully. This could be a handy feature for debugging which exactly part of a consumer-producer queue is failing.

Any chance to get help with that? The it.only() test fails. :\

oleksiyk commented 8 years ago

I wonder why you need to do this at all, topic and partition are available as result[0].topic and result[0].partition here: https://github.com/oleksiyk/kafka/blob/master/test/01.producer.js#L131

koresar commented 8 years ago

DevOps had that error in a service logs. 

2016-10-20T04:08:39.098Z WARN 192-168-1-80.tpgi.com.au Handler for audit-modified:0 failed with 
{ [KafkaError: This request is for a topic or partition that does not exist on this broker.]
  name: 'KafkaError',
  code: 'UnknownTopicOrPartition',
  message: 'This request is for a topic or partition that does not exist on this broker.' }

Which they read as "Unknown topic 'audit-modified'".

They investigated the audit-modified queue back and forth. Found nothing.

Then they googled the error. It lead them to this module. They contacted us. We started an investigation. Then we searched the module source code. And found that consumers are fine. It's just the DevOps didn't understood that the producer is failing. DevOps didn't know the service also can produce, not only consume messages. We explained that they have to create another topic 'critical-change' in the Kafka cluster.

So, this very long story could have been finished much-much quicker by DevOps themselves if they saw something like:

2016-10-20T04:08:39.098Z WARN 192-168-1-80.tpgi.com.au Handler for audit-modified:0 failed with
 { [KafkaError: This request is for a topic or partition that does not exist on this broker.]
  name: 'KafkaError',
  code: 'UnknownTopicOrPartition',
  topic: 'critical-change',
  message: 'This request is for a topic or partition that does not exist on this broker.' }

Which reads as "Unknown topic 'critical-change'".

oleksiyk commented 8 years ago

So just catch errors in your handler and log them with all the info you need. The handler is your own code, it just happened that it also used no-kafka to produce messages to another topic. Just catch errors in your handler.

koresar commented 8 years ago

Thanks. Let me try that.