search

zio / zio-kafka

A Kafka client for ZIO and ZIO Streams
https://zio.dev/zio-kafka
Apache License 2.0
333 stars 134 forks source link

Producer and consumer diagnostics #1039

Open soujiro32167 opened 10 months ago

soujiro32167 commented 10 months ago

I'd like to create middleware for producers and consumers, to do things like distributed tracing, logging, metrics and so on.

Here is an example of instrumenting a producer and consumer with a variation of trace4cats

As you can see, its pretty clunky: I have to backfill all the aliases. It would be nice to be able to do something like

val tracedProducer = producer
  .tapChunk(c => onChunkProduced(c))
  .tapAck(a => onAckReceived(a))

val tracedCosumer = consumer
  .tapChunk(c => onChunkConsumed(c))
  .tapCommit(c => onCommit(c))
svroonland commented 8 months ago

See #1048

erikvanoosten commented 8 months ago

@svroonland We do not actually have diagnostics for producers yet and as far as I know there is no issue requesting those. Should we keep this one open?

svroonland commented 3 months ago

@soujiro32167 This could be of interest for you https://github.com/pierangeloc/zio-theseus/tree/main/modules/tracing-kafka/src/main/scala/io/tuliplogic/ziotoolbox/tracing/kafka

svroonland commented 3 months ago

Also we have consumer metrics since v2.7.3