Allow searching in all partitions of a given event type

[i-am-the-slime]

I really like the search feature, however it becomes tedious very quickly when looking for events in event-types with larger number of partitions. I don't know how the search is implemented currently (low-level API?) but it should be possible to search across partitions.

[SergKam]

We discussed this idea. We should be careful with the implementation. This can kill Nakadi performance if done wrong. Basically, we need to download(consume) to the client the whole data from all partitions as there is no internal server-side search in Kafka. As I see the implementation now, as most probably people want to know the latest/newest events, so The UI should download last 1000 from first all partitions. If the search word is not found, move cursors back on 1000 and download next chanks, and so on. It can be done using low-level API or temporary subscription. The UI should show progress, results, and have the stop/abort button. It would be more efficient to filter data on Nakadi side, at least on Nakadi UI backend side, as the most traffic will happen inside AWS. It means we need workers/jobs infrastructure and progress reporting protocol (pooling, web sockets, long pools, etc)

[rcillo]

When it comes to technical solution, I would say that Nakadi (Kafka) is optimize for sequential reading and not for random access or indexing things. To "pick" an event is a different access pattern. Each storage excels at a serving a different access pattern. If we are to provide this feature, we would benefit from a different storage. The data lake already allows to run arbitrary queries and scan all the data. It's very powerful but takes up to 5 minutes for data to be available there. If the requirement is to have it faster, we would have to keep some sort of "key/value cache". Loading all data from Nakadi (Kafka) on a per request basis is something that would not scale (for very busy event types that means loading gigabytes of data to find a single event).

So, maybe Nakadi UI search could be integrated with some jdbc connector in the data lake? Adding a lookup storage would be quite costly and we should do it only if the data lake does not support this use case already.

[SergKam]

I guess the idea is not to find the event by itself but more to debug problems with nakadi ie confirm that event can be found in Nakadi. So Datalake can be used but it has no guaranty that event is actually in nakadi(error in pipeline or compaction). it makes sense to use datalake for really big event types, for smaller ET we can consume directly from partitions (controlling the amount of data consumed).