search

lensesio / kafka-topics-ui

Web Tool for Kafka Topics |
https://lenses.io
877 stars 148 forks source link

Topics UI not showing records from Kafka 2.0 #133

Closed maver1ck closed 6 years ago

maver1ck commented 6 years ago

This is very strange bug. For some topics I can't see any records and I'm getting "Topic is empty" message.

I increased MAX_BYTES and RECORD_POLL_TIMEOUT, but without success. What I'm doing wrong ?

Docker configuration

KAFKA_REST_PROXY_URL: "http://kafka-cp-kafka-rest-headless:8082"                                                                                                                             
PROXY: true                                                                                                                                                                                  
MAX_BYTES: 500000                                                                                                                                                                            
RECORD_POLL_TIMEOUT: 50000                                                                                                                                                                   
DEBUG_LOGS_ENABLED: true

Logs from UI:

100.96.4.152 - - [08/Oct/2018:13:03:23 +0000] "GET /api/kafka-rest-proxy/topics/topic_v1/partitions HTTP/1.1" 200 169
100.96.4.152 - - [08/Oct/2018:13:03:23 +0000] "POST /api/kafka-rest-proxy/consumers/kafka_topics_ui_avro_2018-10-08-03-03-23-761/instances/kafka-topics-ui-avro/assignments HTTP/1.1" 204 0
100.96.4.152 - - [08/Oct/2018:13:03:24 +0000] "POST /api/kafka-rest-proxy/consumers/kafka_topics_ui_avro_2018-10-08-03-03-23-761/instances/kafka-topics-ui-avro/positions/beginning HTTP/1.1" 204 0
100.96.4.152 - - [08/Oct/2018:13:03:25 +0000] "GET /api/kafka-rest-proxy/consumers/kafka_topics_ui_avro_2018-10-08-03-03-23-761/instances/kafka-topics-ui-avro/records?timeout=50000&max_bytes=500000 HTTP/1.1" 200 2
100.96.4.152 - - [08/Oct/2018:13:03:25 +0000] "DELETE /api/kafka-rest-proxy/consumers/kafka_topics_ui_avro_2018-10-08-03-03-23-761/instances/kafka-topics-ui-avro HTTP/1.1" 204 0

Logs from REST:

[2018-10-08 13:03:23,890] INFO 100.96.19.9 - - [08/Oct/2018:13:03:23 +0000] "GET /topics/topic_v1/partitions HTTP/1.1" 200 169  28 (io.confluent.rest-utils.requests)
[2018-10-08 13:03:23,994] INFO 100.96.19.9 - - [08/Oct/2018:13:03:23 +0000] "POST /consumers/kafka_topics_ui_avro_2018-10-08-03-03-23-761/instances/kafka-topics-ui-avro/assignments HTTP/1.1" 204 0  18 (io.confluent.rest-utils.requests)
[2018-10-08 13:03:24,085] INFO 100.96.19.9 - - [08/Oct/2018:13:03:24 +0000] "POST /consumers/kafka_topics_ui_avro_2018-10-08-03-03-23-761/instances/kafka-topics-ui-avro/positions/beginning HTTP/1.1" 204 0  13 (io.confluent.rest-utils.requests)
[2018-10-08 13:03:24,417] INFO Cluster ID: FEqIfZ57RgOGNNH0K6JH1g (org.apache.kafka.clients.Metadata)
[2018-10-08 13:03:24,719] INFO [Consumer clientId=consumer-10, groupId=kafka_topics_ui_avro_2018-10-08-03-03-23-761] Resetting offset for partition omarsys_customer_v1-0 to offset 0. (org.apache.kafka.clients.consumer.internals.Fetcher)
[2018-10-08 13:03:25,192] INFO 100.96.19.9 - - [08/Oct/2018:13:03:24 +0000] "GET /consumers/kafka_topics_ui_avro_2018-10-08-03-03-23-761/instances/kafka-topics-ui-avro/records?timeout=50000&max_bytes=500000 HTTP/1.1" 200 2  1010 (io.confluent.rest-utils.requests)
[2018-10-08 13:03:25,263] INFO 100.96.19.9 - - [08/Oct/2018:13:03:25 +0000] "DELETE /consumers/kafka_topics_ui_avro_2018-10-08-03-03-23-761/instances/kafka-topics-ui-avro HTTP/1.1" 204 0  4 (io.confluent.rest-utils.requests)

Data is there:

kafka-avro-console-consumer --bootstrap-server kafka-cp-kafka-headless:9092 --topic otopic_v1 --from-beginning --timeout-ms 2000 --max-messages 1 --property schema.registry.url="http://kafka-cp-schema-registry:8081"
{...record...}
Processed a total of 1 messages
andmarios commented 6 years ago

Hi @maver1ck. Our fast-data-dev image offers a Kafka 2.0 tag (landoop/fast-data-dev:2.0) and there Topics UI works.

So I guess the issue may come from your Kafka REST setup. You can try to set consumer.request.timeout.ms to 30 seconds for Kafka REST and see what happens.

maver1ck commented 6 years ago

@andmarios Thank you for showing me this configuration option. That was the reason of my problems.

andmarios commented 6 years ago

Glad we tackled it! Please do not try to set it too low, because of a long-standing bug in Kafka REST, this option controls both the REST's Kafka Consumer and the timeout for the REST clients (like Topics UI).

So if you decrease it to make the REST clients get responses faster, you risk the consumer timing out before consuming messages.

maver1ck commented 6 years ago

I set: for Topics UI MAX_BYTES: 50000
RECORD_POLL_TIMEOUT: 5000 for Kafka REST KAFKA_REST_CONSUMER_REQUEST_TIMEOUT_MS: 30000

It's working fine.