search

dpkp / kafka-python

Python client for Apache Kafka
http://kafka-python.readthedocs.io/
Apache License 2.0
5.6k stars 1.4k forks source link

log configuration #230

Closed hzshlomi closed 10 years ago

hzshlomi commented 10 years ago

a Question: How is the log configured? don't see where handlers are made and attached.... trying to gather some input for exceptions I get and can't find the log file...

dpkp commented 10 years ago

client side logger is 'kafka'. which you should be able configure via

import logging
import sys
logger = logging.getLogger('kafka')
logger.addHandler(logging.StreamHandler(sys.stdout))
logger.setLevel(logging.DEBUG)
hzshlomi commented 10 years ago

10x. did just that since saw no othe roption and I did get logs. What I see now though is a couple of issues:

  1. I see bursts of exceptions trying to send messages using the keyed producer.
  2. I see connection reinitializtions still investigating. HOw do I make the send queue bigger?
dpkp commented 10 years ago

async producer is known to be error prone and not reliable. it does not react well to metadata changes. not sure if that's what you are seeing. You might consider joining the #kafka-python IRC chat on irc.freenode.net to get more interactive help.

wizzat commented 10 years ago

What kind of errors are you seeing? Is your kafka tier going down? Are you having network errors?

hzshlomi commented 10 years ago

I see :

ERROR:kafka:Unable to send messages Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/kafka_python-0.9.2-py2.6.egg/kafka/producer.py", line 188, in send_messages timeout=self.ack_timeout) File "/usr/lib/python2.6/site-packages/kafka_python-0.9.2-py2.6.egg/kafka/client.py", line 317, in send_produce_request self._raise_on_response_error(resp) File "/usr/lib/python2.6/site-packages/kafka_python-0.9.2-py2.6.egg/kafka/client.py", line 189, in _raise_on_response_error kafka.common.check_error(resp) File "/usr/lib/python2.6/site-packages/kafka_python-0.9.2-py2.6.egg/kafka/common.py", line 203, in check_error raise error(response) RequestTimedOutError: ProduceResponse(topic='vpq_android_gcm_h', partition=13, error=7, offset=545140) DEBUG:kafka:Reading response 11218 from Kafka ERROR:kafka:We blew something up: ProduceResponse(topic='vpq_android_gcm_h', partition=13, error=7, offset=545140)

eventually, yes the broker goes down, sometimes with a friend broker too

On Thu, Sep 4, 2014 at 10:05 PM, Mark Roberts notifications@github.com wrote:

What kind of errors are you seeing? Is your kafka tier going down? Are you having network errors?

— Reply to this email directly or view it on GitHub https://github.com/mumrah/kafka-python/issues/230#issuecomment-54528438.

wizzat commented 10 years ago

If your brokers are going down and kafka-python is receiving timeout errors, you may have an issue with your underlying configuration or network topology. You may want to hit up the kafka mailing list.

-Mark

On Sep 6, 2014, at 9:22, hzshlomi notifications@github.com wrote:

I see :

  • DEBUG:kafka:Reinitializing socket connection for shlomi-kafka-broker-3:9092 ============================================================
  • once every few seconds ERROR:kafka:Unable to send messages Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/kafka_python-0.9.2-py2.6.egg/kafka/producer.py", line 188, in send_messages timeout=self.ack_timeout) File "/usr/lib/python2.6/site-packages/kafka_python-0.9.2-py2.6.egg/kafka/client.py", line 317, in send_produce_request self._raise_on_response_error(resp) File "/usr/lib/python2.6/site-packages/kafka_python-0.9.2-py2.6.egg/kafka/client.py", line 189, in _raise_on_response_error kafka.common.check_error(resp) File "/usr/lib/python2.6/site-packages/kafka_python-0.9.2-py2.6.egg/kafka/common.py", line 203, in check_error raise error(response) RequestTimedOutError: ProduceResponse(topic='vpq_android_gcm_h', partition=13, error=7, offset=545140) DEBUG:kafka:Reading response 11218 from Kafka ERROR:kafka:We blew something up: ProduceResponse(topic='vpq_android_gcm_h', partition=13, error=7, offset=545140)

ERROR:kafka:Unable to send messages Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/kafka_python-0.9.2-py2.6.egg/kafka/producer.py", line 188, in send_messages timeout=self.ack_timeout) File "/usr/lib/python2.6/site-packages/kafka_python-0.9.2-py2.6.egg/kafka/client.py", line 317, in send_produce_request self._raise_on_response_error(resp) File "/usr/lib/python2.6/site-packages/kafka_python-0.9.2-py2.6.egg/kafka/client.py", line 189, in _raise_on_response_error kafka.common.check_error(resp) File "/usr/lib/python2.6/site-packages/kafka_python-0.9.2-py2.6.egg/kafka/common.py", line 203, in check_error raise error(response) RequestTimedOutError: ProduceResponse(topic='vpq_android_gcm_h', partition=13, error=7, offset=545140) DEBUG:kafka:Reading response 11218 from Kafka ERROR:kafka:We blew something up: ProduceResponse(topic='vpq_android_gcm_h', partition=13, error=7, offset=545140)

eventually, yes the broker goes down, sometimes with a friend broker too

On Thu, Sep 4, 2014 at 10:05 PM, Mark Roberts notifications@github.com wrote:

What kind of errors are you seeing? Is your kafka tier going down? Are you having network errors?

— Reply to this email directly or view it on GitHub https://github.com/mumrah/kafka-python/issues/230#issuecomment-54528438.

— Reply to this email directly or view it on GitHub.

hzshlomi commented 10 years ago

Yup. I dig that too... :) But maybe there is some value to this list too...

On Sat, Sep 6, 2014 at 7:47 PM, Mark Roberts notifications@github.com wrote:

If your brokers are going down and kafka-python is receiving timeout errors, you may have an issue with your underlying configuration or network topology. You may want to hit up the kafka mailing list.

-Mark

On Sep 6, 2014, at 9:22, hzshlomi notifications@github.com wrote:

I see :

  • DEBUG:kafka:Reinitializing socket connection for shlomi-kafka-broker-3:9092

  • once every few seconds ERROR:kafka:Unable to send messages Traceback (most recent call last): File

"/usr/lib/python2.6/site-packages/kafka_python-0.9.2-py2.6.egg/kafka/producer.py",

line 188, in send_messages timeout=self.ack_timeout) File

"/usr/lib/python2.6/site-packages/kafka_python-0.9.2-py2.6.egg/kafka/client.py",

line 317, in send_produce_request self._raise_on_response_error(resp) File

"/usr/lib/python2.6/site-packages/kafka_python-0.9.2-py2.6.egg/kafka/client.py",

line 189, in _raise_on_response_error kafka.common.check_error(resp) File

"/usr/lib/python2.6/site-packages/kafka_python-0.9.2-py2.6.egg/kafka/common.py",

line 203, in check_error raise error(response) RequestTimedOutError: ProduceResponse(topic='vpq_android_gcm_h', partition=13, error=7, offset=545140) DEBUG:kafka:Reading response 11218 from Kafka ERROR:kafka:We blew something up: ProduceResponse(topic='vpq_android_gcm_h', partition=13, error=7, offset=545140)

ERROR:kafka:Unable to send messages Traceback (most recent call last): File

"/usr/lib/python2.6/site-packages/kafka_python-0.9.2-py2.6.egg/kafka/producer.py",

line 188, in send_messages timeout=self.ack_timeout) File

"/usr/lib/python2.6/site-packages/kafka_python-0.9.2-py2.6.egg/kafka/client.py",

line 317, in send_produce_request self._raise_on_response_error(resp) File

"/usr/lib/python2.6/site-packages/kafka_python-0.9.2-py2.6.egg/kafka/client.py",

line 189, in _raise_on_response_error kafka.common.check_error(resp) File

"/usr/lib/python2.6/site-packages/kafka_python-0.9.2-py2.6.egg/kafka/common.py",

line 203, in check_error raise error(response) RequestTimedOutError: ProduceResponse(topic='vpq_android_gcm_h', partition=13, error=7, offset=545140) DEBUG:kafka:Reading response 11218 from Kafka ERROR:kafka:We blew something up: ProduceResponse(topic='vpq_android_gcm_h', partition=13, error=7, offset=545140)

eventually, yes the broker goes down, sometimes with a friend broker too

On Thu, Sep 4, 2014 at 10:05 PM, Mark Roberts notifications@github.com

wrote:

What kind of errors are you seeing? Is your kafka tier going down? Are you having network errors?

— Reply to this email directly or view it on GitHub < https://github.com/mumrah/kafka-python/issues/230#issuecomment-54528438>.

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub https://github.com/mumrah/kafka-python/issues/230#issuecomment-54720193.

dpkp commented 10 years ago

the except there is RequestTimedOutError on a ProduceRequest. That is caused when the server cannot satisfy your ack_req policy within the time set by ack_timeout. By default kafka-python tells the server to timeout if it can't write the messages to local disk within 1 second. Those settings can be modified when initializing your SimpleProducer on the kafka-python side. But the error typically signals a deeper issue with your brokers.

Closing this issue -- pls reopen if you have further qs. We may do a little work to improve the logging so it is easier to see that the error was caused by an ACK timeout.