search

lensesio / kafka-topics-ui

Web Tool for Kafka Topics |
https://lenses.io
877 stars 148 forks source link

TopicsUI doesn't fetch messages the same way than Kafka CLI? #159

Closed sneko closed 4 years ago

sneko commented 4 years ago

Hi,

When using TopicsUI, for a specific topic the UI will show me 10 pages of messages (even if let's say the last shown offset is around 150), but there is actually a lot more messages (up to offset 300). In this case if I use "seek offset" by setting "150", it will look for the remaining messages.

Ok great "maybe TopicsUI just uses a maximum pagination of 10 pages".

But for another topic, it only gets 4 pages (38 messages fetched) whereas my topic has more than 2000 messages. And in this case the seek offset input doesn't work to fetch an higher range of offsets.

I thought my settings were not pimped enough so I tried this configuration: https://github.com/lensesio/kafka-topics-ui/issues/133#issuecomment-428309506

... but nothing change 😢 . Note that my topics only have 1 partition while testing.

@jglambed any idea what could be the problem? That's quite annoying and I was not able to figure it out for months haha ^^...

Thank you,

mariosant commented 4 years ago

Hey @sneko, TopicsUI uses Confluent REST Proxy API specification to fetch data. The pagination applies for the fetched data set, which is set by the offset parameter, indeed. It can handle larger data sets and you can see this working at our demo setup.

I understand that is not very intuitive - however, we don't have an ETA fixing this.