a kafka plugin for exex

[bowenli86]

Describe the feature

have documented my thoughts in https://x.com/zk_bowen/status/1787248924090351944 and discussed with @gakonst

Motivation

blockchain data indexing is a common need for offchain processing and analytics. It's a common practice to ingest blocks/events into Apache Kafka, the most popular pubsub/message bus system, and then propagate the data to downstream applications. Thus it's great to have a kafka exex built for reth.

i believe transporting the logs/notification to a scalable web2 pubsub system is the most ideal way for downstream processing

• clear separation of concerns. isolated backfill from pubsub instead of reth which won't require re-scanning reth state and cause noisy neighbor to other threads
• richer data engineering ecosystem. with more integrations and toolings from web2. eg, connect kafka/kinesis to a data lake and, boom! you hv a ethereum dataset automatically
• enable quick iteration. reth plugin is still static loaded (at least rn)

Scope

i think it will be of great interest to build following as exex:

core scope

• a kafka exex that supports kafka version greater than 3.3, in semantics of exactly-once message delivery guarantee

strecth scope

• a generelized connector framework for ExEx that keeps pushing ExEx notifcation to popular pubsub infras with sample impls like kafka/kinesis
• a message interface and binary format of ExEx log notification which can be generically consumed and shared among modern pubsubs

Though never coded in rust, i'd love to squeeze out some time and give it a try, with help and support from reth team. Ideally i'd get a mentor from reth's exex developers

Additional context

No response

[onbjerg]

imo this should probably be a separate project, ie. not mainlined, cc @gakonst

[github-actions[bot]]

This issue is stale because it has been open for 21 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 7 days since being marked as stale.