search

ciena / afkak

Kafka client written in Twisted Python
Apache License 2.0
32 stars 22 forks source link

Consumer: max fetch time defaults don't match #99

Open twm opened 5 years ago

twm commented 5 years ago

The Consumer class's fetch_max_wait_time parameter defaults to FETCH_MAX_WAIT_TIME = 100 milliseconds:

https://github.com/ciena/afkak/blob/b7e0a716881ff776e15843465586a4cbd942cb6c/afkak/consumer.py#L46-L47

Internally, Consumer calls KafkaClient.send_fetch_request(). This method has a max_wait_time parameter with a default value of KafkaClient.DEFAULT_FETCH_SERVER_WAIT_MSECS: https://github.com/ciena/afkak/blob/b7e0a716881ff776e15843465586a4cbd942cb6c/afkak/client.py#L151-L158

It doesn't make sense that there are two sets of defaults. It doesn't make sense that they differ.

The defaults in KafkaClient probably affect 0 real users, since everyone uses the Consumer wrapper. They don't really make sense, either. If you want to reduce polling overhead by increasing the fetch long poll period you probably want to reduce the fetch size to something like 1, lest you add tons of latency when the message rate is low.

twm commented 5 years ago

There's another set of defaults on KafkaCodec.encode_fetch_request():

https://github.com/ciena/afkak/blob/b7e0a716881ff776e15843465586a4cbd942cb6c/afkak/kafkacodec.py#L361-L362