tx indexer: add apache kafka event sink

tendermint / tendermint

⟁ Tendermint Core (BFT Consensus) in Go

https://tendermint.com/

Apache License 2.0

5.72k stars 2.07k forks source link

tx indexer: add apache kafka event sink #6584

Closed cmwaters closed 1 year ago

cmwaters commented 3 years ago

Summary

ADR 065 introduced custom TX indexing via event sinks. We also added postgres support which offers a rich query engine for clients (block providers / wallets) that want to make funky transaction queries. The other use case that clients may want transactional data is via event streaming.

Proposal

Apache Kafka is a commonly used tool that allows clients to subscribe to events. We should allow operators to hook up kakfa servers (producers) to a node via the EventSink. I'm not sure if we have to set up the topics or wether the operator has to but I imagine they just correlate to the Event.Type.

For Admin Use

[ ] Not duplicate issue
[ ] Appropriate labels applied
[ ] Appropriate contributors tagged
[ ] Contributor assigned/self-assigned

laniehei commented 3 years ago

A few questions came up for me while thinking about this issue

There are several prominent opensourced kafka go packages available, namely kafka-go, confluent's kafka implementation, and sarama. Is it dealer's choice which one to choose?
Why are several of the methods no-op in the postgres implementation? I couldn't find this in the docs

I have a skeleton implementation, but these questions stopped me up a bit from going further ! tia

alexanderbez commented 3 years ago

Why are several of the methods no-op in the postgres implementation? I couldn't find this in the docs

The no-op methods are the searching methods. The reason they are no-op is because the underlying sink, e.g. postgres, already provides this functionality as that is the entire point of the sink. e.g. you wouldn't provide a SQL query directly to the RPC method, you'd use Postgres directly. The same would apply for Kafka or any other sink.

JayT106 commented 2 years ago

The implementation is quite simple, but I am running into this issue when writing the integrating tests. https://github.com/segmentio/kafka-go/issues/684. Will try sarama package.