Closed lwu08 closed 6 years ago
magnusmaki
commented
6 years ago I have 0.9.4 and am experiencing similar issue... I increased MAX_BYTES to 100.000 but I still only get around 200 messages per topic, and only the oldest messages. I have about 360 topics. Kafka is working fine but it would be nice to have a working ui to it.
magnusmaki
commented
6 years ago Ok hmmm saw now that increasing MAX_BYTES can have been my problem :) I had to increase it to 1000000000 and then some refreshes later in the browser I get everything... :)
mikeghen
commented
6 years ago Is there a way we can have this limit and look at the newest messages?
andmarios
commented
6 years ago You can't because of (a) how Kafka works, (b) how REST proxy works, (c) how Kafka Connect UI works.
a. Kafka semantics have 2 modes for the consumer: earliest offset (what we do currently) and latest, which only shows new messages (not recent/newest, just new) b. REST Proxy uses this semantics, so it can't be asked to fetch you recent messages c. Kafka Connect UI isn't architectured for live streaming of new messages
We offer all these in our enterprise offering, Lenses, which uses its own engine for data browsing and live streaming of new messages.
topicUI version: Running docker KAFKA-TOPICS-UI v0.9.3.
Issue description: My application continuously writes records to a topic. Observe the topic records in topicUI. It seems that topicUI can only show a certain number of records in the topic. After reaching the upper limit, the records shown by topicUI will not extend any more even thought new records are written to the topic. It is reasonable to cut off due to memory/disk resource control. But is there any way to enlarge this upper limit, or possible to list latest records (could overwrite the earlier records)? This is quite useful for monitoring long run topic. Thanks!