search

twmb / franz-go

franz-go contains a feature complete, pure Go library for interacting with Kafka from 0.8.0 through 3.7+. Producing, consuming, transacting, administrating, etc.
BSD 3-Clause "New" or "Revised" License
1.78k stars 182 forks source link

question about sync=>async=>sync messaging #733

Closed vtolstov closed 4 months ago

vtolstov commented 4 months ago

Hi. I have some proxy that receives sync messages (rest/grpc) and sent to backend via kafka. Responses get via single topic and sent back to clients. Now i need to increase number of proxies to 3-8 instances and try to understand how to get response back to needed node that handles client connection. Firstly i'm try to publish via message header kafka partitions assigned to instance that receives client connection. So then somebody wants to send response back to instance response topic it can publish to desired partition.

Does it possible to create something smarter?

twmb commented 4 months ago

Hi there -- unfortunately, I don't really understand the request here. But, my guess is: you can even have a separate response topic, and you can leverage record headers. In the response message, you can put the requesting partition and requesting offset in the response record's headers. Then on the edge when consuming the response topic, you can filter for: I produced to partition 3 at offset 10, now I am looking for a response that contains partition 3 offset 10 in the headers.

Hope this helps, closing for now.