Q: transactional producer configs for mulit-broker(cluster) behind load balancer

[gelihu]

Description

I am trying following:

• Setup Two brokers behind a load balancer(which will route traffic based on availability, load, etc).
• Create a producer(with transaction support) by setting load balancer host address as bootstrap server.
• It works fine fine till there are no disconnects, but one fine day following happens:

%6|1591244167.286|FAIL|app-trx#producer-2| [thrd:sasl_plaintext://load-balanacer-host:9092/bootstrap]: sasl_plaintext://load-balanacer-host:9092/bootstrap: Disconnected (after 1516513ms in state UP)
%3|1591244167.286|ERROR|app-trx#producer-2| [thrd:sasl_plaintext://load-balanacer-host:9092/bootstrap]: sasl_plaintext://load-balanacer-host:9092/bootstrap: Disconnected (after 1516513ms in state UP)
%3|1591244453.733|FAIL|app-trx#producer-2| [thrd:sasl_plaintext://broker-1-ip:9092/1001]: sasl_plaintext://broker-1-ip:9092/1001: Connect to ipv4#broker-1-ip:9092 failed: No route to host (after 3006ms in state CONNECT)
%3|1591244453.733|ERROR|app-trx#producer-2| [thrd:sasl_plaintext://broker-1-ip:9092/1001]: sasl_plaintext://broker-1-ip:9092/1001: Connect to ipv4#broker-1-ip:9092 failed: No route to host (after 3006ms in state CONNECT)
%3|1594301839.904|ADDPARTS|app-trx#producer-2| [thrd:main]: TxnCoordinator/1002: Failed to add partition "topic-name" [0] to transaction: Broker: Producer attempted to use a producer id which is not currently assigned to its transactional id
%1|1594301839.904|TXNERR|app-trx#producer-2| [thrd:main]: Fatal transaction error: Failed to add partitions to transaction: Broker: Producer attempted to use a producer id which is not currently assigned to its transactional id (INVALID_PRODUCER_ID_MAPPING)
%3|1594301839.904|ERROR|app-trx#producer-2| [thrd:main]: Fatal error: Broker: Producer attempted to use a producer id which is not currently assigned to its transactional id: Failed to add partitions to transaction: Broker: Producer attempted to use a producer id which is not currently assigned to its transactional id

Did I configure something wrong, please advice?

I did get on zkCli for /brokers/topics/topic-name, it gave following:

{"version":1,"partitions":{"1":[1001,1002],"0":[1002,1001]}}

How to reproduce

Not an issue.

Checklist

Please provide the following information:

• [ ] A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
• [x] Confluent.Kafka nuget version. 1.4.0
• [ ] Apache Kafka version.
• [ ] Client configuration.
• [ ] Operating system.
• [ ] Provide logs (with "debug" : "..." as necessary in configuration).
• [ ] Provide broker log excerpts.
• [ ] Critical issue.

[mhowlett]

Setup Two brokers behind a load balancer(which will route traffic based on availability, load, etc).

Kafka + protocol take care of availability / load balancing for you already - you shouldn't try to use a load balancer for this purpose. The .net client expects to maintain connections directly to brokers as it requires. I'm aware of some people using a level of indirection on the bootstrap servers to allow them to switch out broker addresses (if all other connections fail, the bootstrap server addresses will be used to find the cluster again), but I haven't done this personally, and I believe there is some devil in the detail here.