Issues with API version negotiation between Kafka producer client and Kafka broker

[adimoraret]

Description

I have encountered some issues with the API version negotiation that the .NET Kafka producer client does it with the Kafka broker. From the logs, it seems the negotiation fails and client gets stuck in a loop re-trying and failing to connect.

How to reproduce

var conf = new ProducerConfig
{
    SecurityProtocol = SecurityProtocol.Plaintext,
    Acks = Acks.All,
    ClientId = "test",
    Debug = "broker,topic,msg",
    BootstrapServers = "b-1.*******************.kafka.us-east-1.amazonaws.com:9092"
};
Action<DeliveryReport<Null, string>> handler = r =>
    Console.WriteLine(!r.Error.IsError
        ? $"Delivered message to {r.TopicPartitionOffset}"
        : $"Delivery Error: {r.Error.Reason}");
using var p = new ProducerBuilder<Null, string>(conf).Build();
p.Produce("test", new Message<Null, string> { Value = "test" }, handler);
p.Flush();

Is there a compatibility issue between this NUGET package version and the Kafka version?

Logs

%7|1705530339.756|NOINFO|test#producer-1| [thrd:main]: Topic test metadata information unknown %7|1705530339.756|NOINFO|test#producer-1| [thrd:main]: Topic test partition count is zero: should refresh metadata %7|1705530339.757|CONNECT|test#producer-1| [thrd:main]: Cluster connection already in progress: refresh unavailable topics %7|1705530339.757|CONNECT|test#producer-1| [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: no cluster connection %7|1705530340.758|NOINFO|test#producer-1| [thrd:main]: Topic test metadata information unknown %7|1705530340.758|NOINFO|test#producer-1| [thrd:main]: Topic test partition count is zero: should refresh metadata %7|1705530340.759|CONNECT|test#producer-1| [thrd:main]: Cluster connection already in progress: refresh unavailable topics %7|1705530340.759|CONNECT|test#producer-1| [thrd:main]: Not selecting any broker for cluster connection: still suppressed for 49ms: no cluster connection %7|1705530341.117|CONNECT|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:90 92/bootstrap: broker in state TRY_CONNECT connecting %7|1705530341.117|STATE|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:9092 /bootstrap: Broker changed state TRY_CONNECT -> CONNECT %7|1705530341.118|CONNECT|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:90 92/bootstrap: Connecting to ipv4#10.227.106.144:9092 (plaintext) with socket 1120 %7|1705530341.122|CONNECT|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:90 92/bootstrap: Connected to ipv4#10.227.106.144:9092 %7|1705530341.122|CONNECTED|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com: 9092/bootstrap: Connected (#8) %7|1705530341.123|FEATURE|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:90 92/bootstrap: Updated enabled protocol features +ApiVersion to ApiVersion %7|1705530341.123|STATE|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:9092 /bootstrap: Broker changed state CONNECT -> APIVERSION_QUERY %7|1705530341.125|FAIL|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:9092/ bootstrap: Disconnected while requesting ApiVersion: might be caused by incorrect security.protocol configuration (connecting to a SSL listener?) or broker version is < 0.10 (see api. version.request) (after 1ms in state APIVERSION_QUERY) (_TRANSPORT): identical to last error: error log suppressed %7|1705530341.125|FEATURE|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:90 92/bootstrap: Updated enabled protocol features -ApiVersion to %7|1705530341.125|STATE|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:9092 /bootstrap: Broker changed state APIVERSION_QUERY -> DOWN %7|1705530341.125|CONNECT|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: Cluster connection already in progress: broker down %7|1705530341.125|STATE|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:9092 /bootstrap: Broker changed state DOWN -> INIT %7|1705530341.125|STATE|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1..kafka.us-east-1.amazonaws.com:9092 /bootstrap: Broker changed state INIT -> TRY_CONNECT %7|1705530341.125|RECONNECT|test#producer-1| [thrd:b-1..kafka.us-east-1.amazonaws.]: b-1.*****.kafka.us-east-1.amazonaws.com: 9092/bootstrap: Delaying next reconnect by 7990ms

Checklist

Please provide the following information:

• [x] A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
• [x] Confluent.Kafka nuget version: 2.3.0
• [x] Apache Kafka version: 2.8.1 (AWS MSK)
• [x] Client configuration: Described above in the coding example
• [x] Operating system: Windows Server 2016 Datacenter
• [x] Provide logs (with "debug" : "..." as necessary in configuration).
• [x] Provide broker log excerpts.
• [ ] Critical issue.

[anchitj]

This mostly means missing configuration. Are you using the correct security.protocol configuration?

[adimoraret]

Yes, it is plaintext.

[anchitj]

@adimoraret Do you have more details on the version of the client you were using? Can you try with the latest version and confirm if it's still an issue?