search

twmb / franz-go

franz-go contains a feature complete, pure Go library for interacting with Kafka from 0.8.0 through 3.8+. Producing, consuming, transacting, administrating, etc.
BSD 3-Clause "New" or "Revised" License
1.85k stars 191 forks source link

Feat request: forwarding records unchanged to a producer client topic #844

Open Nesze opened 1 month ago

Nesze commented 1 month ago

We have many use cases where we consume a record from topic A and forward it unchanged to topic B (in some cases even to separate clusters), essentially relaying it.

This means in our handlers we need to set the consumed record topic to the actual target topic first, can't just blindly forward, e.g.

func(ctx context.Context, r kgo.Record) error {
    r.Topic = produceTopic
    return producer.Produce(ctx, r)
}

It would be handy to be able to configure the producer client to always produce to a given topic. Making r.Topic = produceTopic redundant.

Also, while the docs are clear about this, for newcomers like us, preferring the record topic over the value provided by kgo.DefaultProduceTopic resulted in forwarding the records back to the original topic producing lots of duplicates : ) when switching to this pkg. Therefore in-house we're considering adding an option like AlwaysUseDefaultProduceTopic for our kgo.Client wrappers to guard us against this in the future.

Has anyone had similar issues? And if yes, how did you handle it? Did you build anything specific? Is there any appetite for adding a similar option to this pkg to tweak the final target topic?

mikeywuu commented 3 weeks ago

I would love to use this library in a POC for a new service which relays records from the consumer assigned topic to a destination topic which's name is build dynamically. Therefore, this is unfortunately also blocking me and I'll closely monitor this issue :)