Description

In a C# Windows service application, await producer.ProduceAsync() never return. but the message have sent and consumer received it. This issue both occurred in Confluent.Kafka nuget package of 0.11.6 & 1.0.0 beta3.

Add: Just tested other version and found:

Confluent.Kafka 0.11.2 & librdkkafka.redist 0.11.1: Works
Confluent.Kafka 0.11.3 & librdkkafka.redist 0.11.3: Works
Confluent.Kafka 0.11.4 & librdkkafka.redist 0.11.4: Does not works
Confluent.Kafka 0.11.5 & librdkkafka.redist 0.11.5: Does not works
Confluent.Kafka 0.11.6 & librdkkafka.redist 0.11.6: Does not works

How to reproduce

var config = new ProducerConfig
 {
      BootstrapServers = "server:9092",
      MessageTimeoutMs = 5000
};
var _producer = new ProducerBuilder<Null, string>(config).Build();

var message = new { Time = DateTime.Now };

LogHelper.Debug("Producer start...");

await _producer.ProduceAsync("myTopic", new Message<Null, string>{ Value = JsonConvert.SerializeObject(message)});

LogHelper.Debug("Producer end...");   //this line never executed

Checklist

Please provide the following information:

[X] Confluent.Kafka nuget version: 0.11.6 & 1.0.0-beta3
[X] Apache Kafka version: 2.11
[X] Client configuration: .NET Framework 4.6.2
[X] Operating system: Windows Server 2016
[ ] Provide logs (with "debug" : "..." as necessary in configuration)
[ ] Provide broker log excerpts
[ ] Critical issue

I'm seeing this issue as well. I await on ProduceAsync and never return. I saw this with the simple Producer/Consumer in the readme as well as following the steps in https://www.hugopicado.com/2017/11/22/getting-started-with-net-core-and-kafka.html. Running kafka/zookeeper through Docker using the compose file from the article on the following dotnet --info:

.NET Core SDK (reflecting any global.json):
 Version:   2.1.403
 Commit:    04e15494b6

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.17134
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\2.1.403\

Host (useful for support):
  Version: 2.1.5
  Commit:  290303f510

.NET Core SDKs installed:
  2.1.202 [C:\Program Files\dotnet\sdk]
  2.1.403 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

Attempted to reproduce this on a different machine and see the same issue. Running both under the dotnet core set up above, as well as under .NET Framework 4.6.1.

This will happen if the broker configuration property advertised.listeners has not been set, but your particular setup requires it to be. If you're running Kafka inside a container / VM (if you're on Windows, I guess quite likely!) it will most likely have a default value which is an inaccessible address and you'll need to set it explicitly.

This will happen if the broker configuration property advertised.listeners has not been set, but your particular setup requires it to be. If you're running Kafka inside a container / VM (if you're on Windows, I guess quite likely!) it will most likely have a default value which is an inaccessible IP address and you'll need to set it explicitly.

Hi mhowlett, thanks for your response.

I checked the broker's configuration (config\server.properties) , the property of advertised.listeners have been set with correct value as:

advertised.listeners=PLAINTEXT://myserver.mydomain.com:9092

BTW: the DNS name of myserver.mydomain.com can be resolved correctly on both broker & client side.

Because Confluent.Kafka 0.11.2/0.11.3 are all works very well, only the later version is not works, so it tends to not be the issue of the broker's configuration?

you control the domain name mydomain.com?

If you set the Debug config property to all and past the results here, we should be able to see what the problem is (use 1.0-beta3).

Also seeing this issue using 1.0.0-beta3 from a windows machine to a kafka cluster on ubuntu 16.04 (no containers). I do see this in the logs:

[thrd:ubunutu1-0-11.domain.local:9092/bootstrap]: ubunutu1-0-11.domain.local:9092/bootstrap: failed: err: Local: Host resolution failure: (errno: Bad address)

Even though I can nslookup and ping that address just fine from the command line. Another point of information is that I have more than one network card and the one that is routed to the kafka network is not the primary one.

@mhowlett myserver.mydomain.com is just for example, I actually used an internal hostname in broker's configuration file..

Here is the debug log for 1.0-beta3 (Producer is in Windows Service, broker is on another Windows VM server):

3/2/2019 7:04:22 PM - Debug WAKEUPFD    [thrd:app]: myserver:9092/bootstrap: Enabled low-latency ops queue wake-ups
3/2/2019 7:04:22 PM - Debug BROKER  [thrd:app]: myserver:9092/bootstrap: Added new broker with NodeId -1
3/2/2019 7:04:22 PM - Debug CONNECT [thrd:app]: myserver:9092/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s))
3/2/2019 7:04:23 PM - Debug INIT    [thrd:app]: librdkafka v1.0.0 (0x10000ff) rdkafka#producer-1 initialized (builtin.features 0xffff, debug 0xffff)
3/2/2019 7:04:24 PM - Debug BRKMAIN [thrd::0/internal]: :0/internal: Enter main broker thread
3/2/2019 7:04:24 PM - Debug CONNECT [thrd:main]: myserver:9092/bootstrap: Selected for cluster connection: no cluster connection (broker has 0 connection attempt(s))
3/2/2019 7:04:25 PM - Debug BRKMAIN [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap: Enter main broker thread
3/2/2019 7:04:25 PM - Debug CONNECT [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap: Received CONNECT op
3/2/2019 7:04:25 PM - Debug STATE   [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap: Broker changed state INIT -> TRY_CONNECT
3/2/2019 7:04:25 PM - Debug BROADCAST   [thrd:myserver:9092/bootstrap]: Broadcasting state change
3/2/2019 7:04:25 PM - Debug CONNECT [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap: broker in state TRY_CONNECT connecting
3/2/2019 7:04:25 PM - Debug STATE   [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap: Broker changed state TRY_CONNECT -> CONNECT
3/2/2019 7:04:25 PM - Debug BROADCAST   [thrd:myserver:9092/bootstrap]: Broadcasting state change
3/2/2019 7:04:25 PM - Debug CONNECT [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap: Connecting to ipv4#10.0.0.6:9092 (plaintext) with socket 6000
3/2/2019 7:04:25 PM - Debug CONNECT [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap: Connected to ipv4#10.0.0.6:9092
3/2/2019 7:04:25 PM - Debug CONNECTED   [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap: Connected (#1)
3/2/2019 7:04:25 PM - Debug FEATURE [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap: Updated enabled protocol features +ApiVersion to ApiVersion
3/2/2019 7:04:25 PM - Debug STATE   [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap: Broker changed state CONNECT -> APIVERSION_QUERY
3/2/2019 7:04:25 PM - Debug BROADCAST   [thrd:myserver:9092/bootstrap]: Broadcasting state change
3/2/2019 7:04:25 PM - Debug SEND    [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap: Sent ApiVersionRequest (v0, 25 bytes @ 0, CorrId 1)
3/2/2019 7:04:25 PM - Debug RECV    [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap: Received ApiVersionResponse (v0, 264 bytes, CorrId 1, rtt 1.45ms)
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap: Broker API support:
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey Produce (0) Versions 0..5
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey Fetch (1) Versions 0..7
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey Offset (2) Versions 0..2
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey Metadata (3) Versions 0..5
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey LeaderAndIsr (4) Versions 0..1
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey StopReplica (5) Versions 0..0
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey UpdateMetadata (6) Versions 0..4
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey ControlledShutdown (7) Versions 0..1
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey OffsetCommit (8) Versions 0..3
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey OffsetFetch (9) Versions 0..3
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey GroupCoordinator (10) Versions 0..1
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey JoinGroup (11) Versions 0..2
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey Heartbeat (12) Versions 0..1
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey LeaveGroup (13) Versions 0..1
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey SyncGroup (14) Versions 0..1
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey DescribeGroups (15) Versions 0..1
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey ListGroups (16) Versions 0..1
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey SaslHandshake (17) Versions 0..1
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey ApiVersion (18) Versions 0..1
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey CreateTopics (19) Versions 0..2
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey DeleteTopics (20) Versions 0..1
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey DeleteRecords (21) Versions 0..0
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey InitProducerId (22) Versions 0..0
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey OffsetForLeaderEpoch (23) Versions 0..0
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey AddPartitionsToTxn (24) Versions 0..0
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey AddOffsetsToTxn (25) Versions 0..0
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey EndTxn (26) Versions 0..0
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey WriteTxnMarkers (27) Versions 0..0
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey TxnOffsetCommit (28) Versions 0..0
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey DescribeAcls (29) Versions 0..0
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey CreateAcls (30) Versions 0..0
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey DeleteAcls (31) Versions 0..0
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey DescribeConfigs (32) Versions 0..1
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey AlterConfigs (33) Versions 0..0
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey AlterReplicaLogDirs (34) Versions 0..0
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey DescribeLogDirs (35) Versions 0..0
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey SaslAuthenticate (36) Versions 0..0
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey CreatePartitions (37) Versions 0..0
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey CreateDelegationToken (38) Versions 0..0
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey RenewDelegationToken (39) Versions 0..0
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey ExpireDelegationToken (40) Versions 0..0
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey DescribeDelegationToken (41) Versions 0..0
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:   ApiKey DeleteGroups (42) Versions 0..0
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:  Feature MsgVer1: Produce (2..2) supported by broker
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:  Feature MsgVer1: Fetch (2..2) supported by broker
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap: Enabling feature MsgVer1
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:  Feature MsgVer2: Produce (3..3) supported by broker
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:  Feature MsgVer2: Fetch (4..4) supported by broker
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap: Enabling feature MsgVer2
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:  Feature ApiVersion: ApiVersion (0..0) supported by broker
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap: Enabling feature ApiVersion
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:  Feature BrokerGroupCoordinator: GroupCoordinator (0..0) supported by broker
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap: Enabling feature BrokerGroupCoordinator
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:  Feature BrokerBalancedConsumer: GroupCoordinator (0..0) supported by broker
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:  Feature BrokerBalancedConsumer: OffsetCommit (1..2) supported by broker
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:  Feature BrokerBalancedConsumer: OffsetFetch (1..1) supported by broker
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:  Feature BrokerBalancedConsumer: JoinGroup (0..0) supported by broker
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:  Feature BrokerBalancedConsumer: SyncGroup (0..0) supported by broker
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:  Feature BrokerBalancedConsumer: Heartbeat (0..0) supported by broker
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:  Feature BrokerBalancedConsumer: LeaveGroup (0..0) supported by broker
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap: Enabling feature BrokerBalancedConsumer
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:  Feature ThrottleTime: Produce (1..2) supported by broker
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:  Feature ThrottleTime: Fetch (1..2) supported by broker
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap: Enabling feature ThrottleTime
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:  Feature Sasl: JoinGroup (0..0) supported by broker
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap: Enabling feature Sasl
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:  Feature SaslHandshake: SaslHandshake (0..0) supported by broker
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap: Enabling feature SaslHandshake
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:  Feature LZ4: GroupCoordinator (0..0) supported by broker
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap: Enabling feature LZ4
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:  Feature OffsetTime: Offset (1..1) supported by broker
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap: Enabling feature OffsetTime
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:  Feature IdempotentProducer: InitProducerId (0..0) supported by broker
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap: Enabling feature IdempotentProducer
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:  Feature ZSTD: Produce (7..7) NOT supported by broker
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap:  Feature ZSTD: Fetch (10..10) NOT supported by broker
3/2/2019 7:04:25 PM - Debug APIVERSION  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap: Disabling feature ZSTD
3/2/2019 7:04:25 PM - Debug FEATURE [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap: Updated enabled protocol features to MsgVer1,ApiVersion,BrokerBalancedConsumer,ThrottleTime,Sasl,SaslHandshake,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2,IdempotentProducer
3/2/2019 7:04:25 PM - Debug STATE   [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap: Broker changed state APIVERSION_QUERY -> UP
3/2/2019 7:04:25 PM - Debug BROADCAST   [thrd:myserver:9092/bootstrap]: Broadcasting state change
3/2/2019 7:04:25 PM - Debug METADATA    [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap: Request metadata for brokers only: connected
3/2/2019 7:04:25 PM - Debug SEND    [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap: Sent MetadataRequest (v2, 25 bytes @ 0, CorrId 2)
3/2/2019 7:04:25 PM - Debug RECV    [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap: Received MetadataResponse (v2, 63 bytes, CorrId 2, rtt 2.69ms)
3/2/2019 7:04:25 PM - Debug METADATA    [thrd:main]: myserver:9092/bootstrap: ===== Received metadata: connected =====
3/2/2019 7:04:25 PM - Debug METADATA    [thrd:main]: myserver:9092/bootstrap: ClusterId: 8bhByu-2R8e_eZrESFTZjw, ControllerId: 0
3/2/2019 7:04:25 PM - Debug METADATA    [thrd:main]: myserver:9092/bootstrap: 1 brokers, 0 topics
3/2/2019 7:04:25 PM - Debug METADATA    [thrd:main]: myserver:9092/bootstrap:   Broker #0/1: myserver:9092 NodeId 0
3/2/2019 7:04:25 PM - Debug UPDATE  [thrd:myserver:9092/bootstrap]: myserver:9092/bootstrap: NodeId changed from -1 to 0
3/2/2019 7:04:25 PM - Debug UPDATE  [thrd:myserver:9092/bootstrap]: myserver:9092/0: Name changed from myserver:9092/bootstrap to myserver:9092/0
3/2/2019 7:04:25 PM - Debug LEADER  [thrd:myserver:9092/bootstrap]: myserver:9092/0: Mapped 0 partition(s) to broker
3/2/2019 7:04:25 PM - Debug STATE   [thrd:myserver:9092/bootstrap]: myserver:9092/0: Broker changed state UP -> UPDATE
3/2/2019 7:04:25 PM - Debug BROADCAST   [thrd:myserver:9092/bootstrap]: Broadcasting state change
3/2/2019 7:04:25 PM - Debug BROADCAST   [thrd:myserver:9092/bootstrap]: Broadcasting state change
3/2/2019 7:04:25 PM - Debug STATE   [thrd:myserver:9092/bootstrap]: myserver:9092/0: Broker changed state UPDATE -> UP
3/2/2019 7:04:25 PM - Debug BROADCAST   [thrd:myserver:9092/bootstrap]: Broadcasting state change
3/2/2019 7:04:25 PM - Debug CLUSTERID   [thrd:main]: myserver:9092/0: ClusterId update "" -> "8bhByu-2R8e_eZrESFTZjw"
3/2/2019 7:04:25 PM - Debug CONTROLLERID    [thrd:main]: myserver:9092/0: ControllerId update -1 -> 0
3/2/2019 7:04:25 PM - Debug BROADCAST   [thrd:main]: Broadcasting state change
3/2/2019 7:04:42 PM - Debug TOPIC   [thrd:app]: New local topic: MailNotification
3/2/2019 7:04:42 PM - Debug TOPPARNEW   [thrd:app]: NEW MailNotification [-1] 00000180FA9F5500 (at rd_kafka_topic_new0:393)
3/2/2019 7:04:42 PM - Debug NOINFO  [thrd:main]: Topic MailNotification metadata information unknown
3/2/2019 7:04:42 PM - Debug NOINFO  [thrd:main]: Topic MailNotification partition count is zero: should refresh metadata
3/2/2019 7:04:42 PM - Debug METADATA    [thrd:main]: Requesting metadata for 1/1 topics: refresh unavailable topics
3/2/2019 7:04:42 PM - Debug METADATA    [thrd:main]: myserver:9092/0: Request metadata for 1 topic(s): refresh unavailable topics
3/2/2019 7:04:42 PM - Debug SEND    [thrd:myserver:9092/bootstrap]: myserver:9092/0: Sent MetadataRequest (v2, 43 bytes @ 0, CorrId 3)
3/2/2019 7:04:42 PM - Debug RECV    [thrd:myserver:9092/bootstrap]: myserver:9092/0: Received MetadataResponse (v2, 114 bytes, CorrId 3, rtt 2.13ms)
3/2/2019 7:04:42 PM - Debug METADATA    [thrd:main]: myserver:9092/0: ===== Received metadata (for 1 requested topics): refresh unavailable topics =====
3/2/2019 7:04:42 PM - Debug METADATA    [thrd:main]: myserver:9092/0: ClusterId: 8bhByu-2R8e_eZrESFTZjw, ControllerId: 0
3/2/2019 7:04:42 PM - Debug METADATA    [thrd:main]: myserver:9092/0: 1 brokers, 1 topics
3/2/2019 7:04:42 PM - Debug METADATA    [thrd:main]: myserver:9092/0:   Broker #0/1: myserver:9092 NodeId 0
3/2/2019 7:04:42 PM - Debug METADATA    [thrd:main]: myserver:9092/0:   Topic #0/1: MailNotification with 1 partitions
3/2/2019 7:04:42 PM - Debug STATE   [thrd:main]: Topic MailNotification changed state unknown -> exists
3/2/2019 7:04:42 PM - Debug PARTCNT [thrd:main]: Topic MailNotification partition count changed from 0 to 1
3/2/2019 7:04:42 PM - Debug TOPPARNEW   [thrd:main]: NEW MailNotification [0] 00000180FA9AAF90 (at rd_kafka_topic_partition_cnt_update:620)
3/2/2019 7:04:42 PM - Debug METADATA    [thrd:main]:   Topic MailNotification partition 0 Leader 0
3/2/2019 7:04:42 PM - Debug BRKDELGT    [thrd:main]: MailNotification [0]: delegate to broker myserver:9092/0 (rktp 00000180FA9AAF90, term 0, ref 2, remove 0)
3/2/2019 7:04:42 PM - Debug BRKDELGT    [thrd:main]: MailNotification [0]: broker myserver:9092/0 is now leader for partition with 0 messages (0 bytes) queued
3/2/2019 7:04:42 PM - Debug BRKMIGR [thrd:main]: Migrating topic MailNotification [0] 00000180FA9AAF90 from (none) to myserver:9092/0 (sending PARTITION_JOIN to myserver:9092/0)
3/2/2019 7:04:42 PM - Debug TOPBRK  [thrd:myserver:9092/bootstrap]: myserver:9092/0: Topic MailNotification [0]: joining broker (rktp 00000180FA9AAF90, 0 message(s) queued)
3/2/2019 7:04:42 PM - Debug FETCHADD    [thrd:myserver:9092/bootstrap]: myserver:9092/0: Added MailNotification [0] to active list (1 entries, opv 0, 0 messages queued)
3/2/2019 7:04:42 PM - Debug BROADCAST   [thrd:myserver:9092/bootstrap]: Broadcasting state change
3/2/2019 7:04:42 PM - Debug PARTCNT [thrd:main]: Partitioning 1 unassigned messages in topic MailNotification to 1 partitions
3/2/2019 7:04:42 PM - Debug TOPPAR  [thrd:myserver:9092/bootstrap]: myserver:9092/0: MailNotification [0] 1 message(s) in xmit queue (1 added from partition queue)
3/2/2019 7:04:42 PM - Debug PRODUCE [thrd:myserver:9092/bootstrap]: myserver:9092/0: MailNotification [0]: Produce MessageSet with 1 message(s) (414 bytes, ApiVersion 5, MsgVersion 2, MsgId 0, BaseSeq -1, PID{Invalid})
3/2/2019 7:04:42 PM - Debug SEND    [thrd:myserver:9092/bootstrap]: myserver:9092/0: Sent ProduceRequest (v5, 477 bytes @ 0, CorrId 4)
3/2/2019 7:04:42 PM - Debug UAS [thrd:main]: 1/1 messages were partitioned in topic MailNotification
3/2/2019 7:04:42 PM - Debug METADATA    [thrd:main]: myserver:9092/0: 1/1 requested topic(s) seen in metadata
3/2/2019 7:04:42 PM - Debug RECV    [thrd:myserver:9092/bootstrap]: myserver:9092/0: Received ProduceResponse (v5, 60 bytes, CorrId 4, rtt 25.32ms)
3/2/2019 7:04:42 PM - Debug MSGSET  [thrd:myserver:9092/bootstrap]: myserver:9092/0: MailNotification [0]: MessageSet with 1 message(s) (MsgId 0, BaseSeq -1) delivered
3/2/2019 7:09:23 PM - Debug METADATA    [thrd:main]: Requesting metadata for 1/1 topics: periodic refresh
3/2/2019 7:09:23 PM - Debug METADATA    [thrd:main]: myserver:9092/0: Request metadata for 1 topic(s): periodic refresh
3/2/2019 7:09:23 PM - Debug SEND    [thrd:myserver:9092/bootstrap]: myserver:9092/0: Sent MetadataRequest (v2, 43 bytes @ 0, CorrId 5)
3/2/2019 7:09:23 PM - Debug RECV    [thrd:myserver:9092/bootstrap]: myserver:9092/0: Received MetadataResponse (v2, 114 bytes, CorrId 5, rtt 2.86ms)
3/2/2019 7:09:23 PM - Debug METADATA    [thrd:main]: myserver:9092/0: ===== Received metadata (for 1 requested topics): periodic refresh =====
3/2/2019 7:09:23 PM - Debug METADATA    [thrd:main]: myserver:9092/0: ClusterId: 8bhByu-2R8e_eZrESFTZjw, ControllerId: 0
3/2/2019 7:09:23 PM - Debug METADATA    [thrd:main]: myserver:9092/0: 1 brokers, 1 topics
3/2/2019 7:09:23 PM - Debug METADATA    [thrd:main]: myserver:9092/0:   Broker #0/1: myserver:9092 NodeId 0
3/2/2019 7:09:23 PM - Debug METADATA    [thrd:main]: myserver:9092/0:   Topic #0/1: MailNotification with 1 partitions
3/2/2019 7:09:23 PM - Debug METADATA    [thrd:main]:   Topic MailNotification partition 0 Leader 0
3/2/2019 7:09:23 PM - Debug METADATA    [thrd:main]: myserver:9092/0: 1/1 requested topic(s) seen in metadata
3/2/2019 7:14:23 PM - Debug METADATA    [thrd:main]: Requesting metadata for 1/1 topics: periodic refresh
3/2/2019 7:14:23 PM - Debug METADATA    [thrd:main]: myserver:9092/0: Request metadata for 1 topic(s): periodic refresh
3/2/2019 7:14:23 PM - Debug SEND    [thrd:myserver:9092/bootstrap]: myserver:9092/0: Sent MetadataRequest (v2, 43 bytes @ 0, CorrId 6)
3/2/2019 7:14:23 PM - Debug RECV    [thrd:myserver:9092/bootstrap]: myserver:9092/0: Received MetadataResponse (v2, 114 bytes, CorrId 6, rtt 2.57ms)
3/2/2019 7:14:23 PM - Debug METADATA    [thrd:main]: myserver:9092/0: ===== Received metadata (for 1 requested topics): periodic refresh =====
3/2/2019 7:14:23 PM - Debug METADATA    [thrd:main]: myserver:9092/0: ClusterId: 8bhByu-2R8e_eZrESFTZjw, ControllerId: 0
3/2/2019 7:14:23 PM - Debug METADATA    [thrd:main]: myserver:9092/0: 1 brokers, 1 topics
3/2/2019 7:14:23 PM - Debug METADATA    [thrd:main]: myserver:9092/0:   Broker #0/1: myserver:9092 NodeId 0
3/2/2019 7:14:23 PM - Debug METADATA    [thrd:main]: myserver:9092/0:   Topic #0/1: MailNotification with 1 partitions
3/2/2019 7:14:23 PM - Debug METADATA    [thrd:main]:   Topic MailNotification partition 0 Leader 0
3/2/2019 7:14:23 PM - Debug METADATA    [thrd:main]: myserver:9092/0: 1/1 requested topic(s) seen in metadata

I'm seeing this as well. I am using Confluent.Kafka version 1.0.0-beta3-ci-861 running on a mac (Mojave) with librdkafka 0.11.6. I can call my kafka when using the command line tools (e.g. kafka-console-producer --broker-list localhost:9092 --topic test) but when I try to do the same with ProduceAsync() it just hangs. No exceptions, no return. Debug "all" for me doesn't show anything either.

thanks for the logs @Li-Yanzhi - they demonstrate this is not due to the advertised.listener scenario i mentioned. First, I would strongly recommend not trying to run the Kafka brokers on Windows. They aren't tested on this platform to my knowledge + I've experienced various issues trying to do this myself for the purposes of testing the .net client, so I would absolutely not trust it. My next step suggestion for you is to run Kafka on a linux VM.

@cmills2000 - does setting an IP address instead of domain name work? do you see any logs saying the leader is down?

@jeffhollan - can you print out the Library.VersionString to verify the version of librdafka being loaded (there are some discovery issues on macos) and make sure you don't have any brew install versions of librdkakfa installed.

I was unable to replicate these issues just now on Windows Pro 10 (.NET Framework, .NET Core). Also, I use await ProduceAsync with 1.0-beta3 on both mac os and linux successfully all the time.

Given the number of people reporting this i'd like to get to the bottom of what's going on, but I don't have any good ideas at the moment.

My Libarary.VesrsionString returns 0.11.6, however I do see I have a version of brew installed librdkafka. When I uninstalled it I get an error that it

Failed to load the librdkafka native library

Let me know if there's a better way to install librdkafka once I remove it from brew. If I need to move to another issue I can do that as well so you can focus on this specific bug if mine may be brew related

@cmills2000 - does setting an IP address instead of domain name work? do you see any logs saying the leader is down?

Hi @mhowlett, I did try with IP's instead of hostnames, but still same issue, looks like a host resolution failure. I will check again to make sure. I don't see anything in the logs that indicate an issue with a leader being down.

As another point of information, two windows updates have been installed recently on my Windows 10 machine (KB4023057 and KB4487017).

I may be way off, but looking at perhaps an issue with System.Net.Dns.BeginGetHostAddresses call as I am having issues with another Kafka library as well (kafka-sharp).

@cmills2000 - Confluent.Kafka doesn't do dns lookups in managed code, the underlying C library libdkafka does all of that.

@mhowlett ok, fair enough. Looks like I can connect now, but the ProduceAsync method still does not return:

[thrd:app]: ubunutu1-0-11.domain.local:9092/bootstrap: Enabled low-latency ops queue wake-ups [thrd::0/internal]: :0/internal: Enter main broker thread [thrd:app]: ubunutu1-0-11.domain.local:9092/bootstrap: Added new broker with NodeId -1 [thrd:app]: ubunutu1-0-12.domain.local:9092/bootstrap: Enabled low-latency ops queue wake-ups [thrd:ubunutu1-0-11.domain.local:9092/bootstrap]: ubunutu1-0-11.domain.local:9092/bootstrap: Enter main broker thread [thrd:app]: ubunutu1-0-12.domain.local:9092/bootstrap: Added new broker with NodeId -1 [thrd:app]: ubunutu1-0-13.domain.local:9092/bootstrap: Enabled low-latency ops queue wake-ups [thrd:app]: ubunutu1-0-13.domain.local:9092/bootstrap: Added new broker with NodeId -1 [thrd:app]: ubunutu1-0-13.domain.local:9092/bootstrap: Selected for cluster connection: bootstrap servers added (broker has 0 connection attempt(s)) [thrd:app]: librdkafka v1.0.0 (0x10000ff) rdkafka#producer-1 initialized (builtin.features 0xffff, debug 0xffff) [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Enter main broker thread [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Enter main broker thread [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Received CONNECT op [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Broker changed state INIT -> TRY_CONNECT [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: Broadcasting state change [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: broker in state TRY_CONNECT connecting [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Broker changed state TRY_CONNECT -> CONNECT [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: Broadcasting state change [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Connecting to ipv4#10.1.0.13:9092 (plaintext) with socket 3956 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Connected to ipv4#10.1.0.13:9092 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Connected (#1) [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Updated enabled protocol features +ApiVersion to ApiVersion [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Broker changed state CONNECT -> APIVERSION_QUERY [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: Broadcasting state change [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Sent ApiVersionRequest (v0, 25 bytes @ 0, CorrId 1) [thrd:main]: ubunutu1-0-12.domain.local:9092/bootstrap: Selected for cluster connection: no cluster connection (broker has 0 connection attempt(s)) [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Received CONNECT op [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Broker changed state INIT -> TRY_CONNECT [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: Broadcasting state change [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: broker in state TRY_CONNECT connecting [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Broker changed state TRY_CONNECT -> CONNECT [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: Broadcasting state change [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Connecting to ipv4#10.1.0.12:9092 (plaintext) with socket 4140 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Connected to ipv4#10.1.0.12:9092 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Connected (#1) [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Updated enabled protocol features +ApiVersion to ApiVersion [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Broker changed state CONNECT -> APIVERSION_QUERY [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: Broadcasting state change [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Received ApiVersionResponse (v0, 264 bytes, CorrId 1, rtt 18.62ms) [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Sent ApiVersionRequest (v0, 25 bytes @ 0, CorrId 1) [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Broker API support: [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey Produce (0) Versions 0..7 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey Fetch (1) Versions 0..10 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey Offset (2) Versions 0..4 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey Metadata (3) Versions 0..7 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey LeaderAndIsr (4) Versions 0..1 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey StopReplica (5) Versions 0..0 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey UpdateMetadata (6) Versions 0..4 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey ControlledShutdown (7) Versions 0..1 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey OffsetCommit (8) Versions 0..6 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey OffsetFetch (9) Versions 0..5 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey GroupCoordinator (10) Versions 0..2 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey JoinGroup (11) Versions 0..3 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey Heartbeat (12) Versions 0..2 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey LeaveGroup (13) Versions 0..2 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey SyncGroup (14) Versions 0..2 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey DescribeGroups (15) Versions 0..2 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey ListGroups (16) Versions 0..2 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey SaslHandshake (17) Versions 0..1 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey ApiVersion (18) Versions 0..2 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey CreateTopics (19) Versions 0..3 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey DeleteTopics (20) Versions 0..3 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey DeleteRecords (21) Versions 0..1 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey InitProducerId (22) Versions 0..1 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey OffsetForLeaderEpoch (23) Versions 0..2 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey AddPartitionsToTxn (24) Versions 0..1 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey AddOffsetsToTxn (25) Versions 0..1 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey EndTxn (26) Versions 0..1 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey WriteTxnMarkers (27) Versions 0..0 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey TxnOffsetCommit (28) Versions 0..2 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey DescribeAcls (29) Versions 0..1 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey CreateAcls (30) Versions 0..1 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey DeleteAcls (31) Versions 0..1 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey DescribeConfigs (32) Versions 0..2 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey AlterConfigs (33) Versions 0..1 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey AlterReplicaLogDirs (34) Versions 0..1 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey DescribeLogDirs (35) Versions 0..1 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey SaslAuthenticate (36) Versions 0..0 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey CreatePartitions (37) Versions 0..1 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey CreateDelegationToken (38) Versions 0..1 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey RenewDelegationToken (39) Versions 0..1 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey ExpireDelegationToken (40) Versions 0..1 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey DescribeDelegationToken (41) Versions 0..1 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: ApiKey DeleteGroups (42) Versions 0..1 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Feature MsgVer1: Produce (2..2) supported by broker [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Feature MsgVer1: Fetch (2..2) supported by broker [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Enabling feature MsgVer1 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Feature MsgVer2: Produce (3..3) supported by broker [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Feature MsgVer2: Fetch (4..4) supported by broker [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Enabling feature MsgVer2 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Feature ApiVersion: ApiVersion (0..0) supported by broker [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Enabling feature ApiVersion [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Feature BrokerGroupCoordinator: GroupCoordinator (0..0) supported by broker [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Enabling feature BrokerGroupCoordinator [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Feature BrokerBalancedConsumer: GroupCoordinator (0..0) supported by broker [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Feature BrokerBalancedConsumer: OffsetCommit (1..2) supported by broker [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Feature BrokerBalancedConsumer: OffsetFetch (1..1) supported by broker [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Feature BrokerBalancedConsumer: JoinGroup (0..0) supported by broker [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Feature BrokerBalancedConsumer: SyncGroup (0..0) supported by broker [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Feature BrokerBalancedConsumer: Heartbeat (0..0) supported by broker [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Feature BrokerBalancedConsumer: LeaveGroup (0..0) supported by broker [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Enabling feature BrokerBalancedConsumer [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Feature ThrottleTime: Produce (1..2) supported by broker [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Feature ThrottleTime: Fetch (1..2) supported by broker [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Enabling feature ThrottleTime [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Feature Sasl: JoinGroup (0..0) supported by broker [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Enabling feature Sasl [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Feature SaslHandshake: SaslHandshake (0..0) supported by broker [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Enabling feature SaslHandshake [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Feature LZ4: GroupCoordinator (0..0) supported by broker [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Enabling feature LZ4 [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Feature OffsetTime: Offset (1..1) supported by broker [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Enabling feature OffsetTime [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Feature IdempotentProducer: InitProducerId (0..0) supported by broker [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Enabling feature IdempotentProducer [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Feature ZSTD: Produce (7..7) supported by broker [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Feature ZSTD: Fetch (10..10) supported by broker [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Enabling feature ZSTD [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Updated enabled protocol features to MsgVer1,ApiVersion,BrokerBalancedConsumer,ThrottleTime,Sasl,SaslHandshake,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2,IdempotentProducer,ZSTD [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Broker changed state APIVERSION_QUERY -> UP [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: Broadcasting state change [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Request metadata for brokers only: connected [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Sent MetadataRequest (v2, 25 bytes @ 0, CorrId 2) [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Received ApiVersionResponse (v0, 264 bytes, CorrId 1, rtt 16.93ms) [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Broker API support: [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey Produce (0) Versions 0..7 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey Fetch (1) Versions 0..10 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey Offset (2) Versions 0..4 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey Metadata (3) Versions 0..7 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey LeaderAndIsr (4) Versions 0..1 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey StopReplica (5) Versions 0..0 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey UpdateMetadata (6) Versions 0..4 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey ControlledShutdown (7) Versions 0..1 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey OffsetCommit (8) Versions 0..6 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey OffsetFetch (9) Versions 0..5 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey GroupCoordinator (10) Versions 0..2 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey JoinGroup (11) Versions 0..3 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey Heartbeat (12) Versions 0..2 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey LeaveGroup (13) Versions 0..2 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey SyncGroup (14) Versions 0..2 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey DescribeGroups (15) Versions 0..2 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey ListGroups (16) Versions 0..2 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey SaslHandshake (17) Versions 0..1 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey ApiVersion (18) Versions 0..2 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey CreateTopics (19) Versions 0..3 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey DeleteTopics (20) Versions 0..3 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey DeleteRecords (21) Versions 0..1 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey InitProducerId (22) Versions 0..1 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey OffsetForLeaderEpoch (23) Versions 0..2 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey AddPartitionsToTxn (24) Versions 0..1 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey AddOffsetsToTxn (25) Versions 0..1 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey EndTxn (26) Versions 0..1 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey WriteTxnMarkers (27) Versions 0..0 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey TxnOffsetCommit (28) Versions 0..2 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey DescribeAcls (29) Versions 0..1 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey CreateAcls (30) Versions 0..1 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey DeleteAcls (31) Versions 0..1 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey DescribeConfigs (32) Versions 0..2 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey AlterConfigs (33) Versions 0..1 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey AlterReplicaLogDirs (34) Versions 0..1 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey DescribeLogDirs (35) Versions 0..1 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey SaslAuthenticate (36) Versions 0..0 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey CreatePartitions (37) Versions 0..1 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey CreateDelegationToken (38) Versions 0..1 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey RenewDelegationToken (39) Versions 0..1 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey ExpireDelegationToken (40) Versions 0..1 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey DescribeDelegationToken (41) Versions 0..1 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: ApiKey DeleteGroups (42) Versions 0..1 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Feature MsgVer1: Produce (2..2) supported by broker [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Feature MsgVer1: Fetch (2..2) supported by broker [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Enabling feature MsgVer1 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Feature MsgVer2: Produce (3..3) supported by broker [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Feature MsgVer2: Fetch (4..4) supported by broker [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Enabling feature MsgVer2 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Feature ApiVersion: ApiVersion (0..0) supported by broker [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Enabling feature ApiVersion [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Feature BrokerGroupCoordinator: GroupCoordinator (0..0) supported by broker [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Enabling feature BrokerGroupCoordinator [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Feature BrokerBalancedConsumer: GroupCoordinator (0..0) supported by broker [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Feature BrokerBalancedConsumer: OffsetCommit (1..2) supported by broker [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Feature BrokerBalancedConsumer: OffsetFetch (1..1) supported by broker [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Feature BrokerBalancedConsumer: JoinGroup (0..0) supported by broker [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Feature BrokerBalancedConsumer: SyncGroup (0..0) supported by broker [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Feature BrokerBalancedConsumer: Heartbeat (0..0) supported by broker [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Feature BrokerBalancedConsumer: LeaveGroup (0..0) supported by broker [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Enabling feature BrokerBalancedConsumer [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Feature ThrottleTime: Produce (1..2) supported by broker [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Feature ThrottleTime: Fetch (1..2) supported by broker [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Enabling feature ThrottleTime [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Feature Sasl: JoinGroup (0..0) supported by broker [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Enabling feature Sasl [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Feature SaslHandshake: SaslHandshake (0..0) supported by broker [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Enabling feature SaslHandshake [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Feature LZ4: GroupCoordinator (0..0) supported by broker [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Enabling feature LZ4 [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Feature OffsetTime: Offset (1..1) supported by broker [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Enabling feature OffsetTime [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Feature IdempotentProducer: InitProducerId (0..0) supported by broker [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Enabling feature IdempotentProducer [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Feature ZSTD: Produce (7..7) supported by broker [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Feature ZSTD: Fetch (10..10) supported by broker [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Enabling feature ZSTD [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Updated enabled protocol features to MsgVer1,ApiVersion,BrokerBalancedConsumer,ThrottleTime,Sasl,SaslHandshake,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2,IdempotentProducer,ZSTD [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Broker changed state APIVERSION_QUERY -> UP [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: Broadcasting state change [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Request metadata for brokers only: connected [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Skipping metadata request: connected: full request already in-transit [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Received MetadataResponse (v2, 183 bytes, CorrId 2, rtt 20.00ms) [thrd:main]: ubunutu1-0-13.domain.local:9092/bootstrap: ===== Received metadata: connected ===== [thrd:main]: ubunutu1-0-13.domain.local:9092/bootstrap: ClusterId: xD4GZxzuQ1mA0xU_4HFxCA, ControllerId: 3 [thrd:main]: ubunutu1-0-13.domain.local:9092/bootstrap: 3 brokers, 0 topics [thrd:main]: ubunutu1-0-13.domain.local:9092/bootstrap: Broker #0/3: ubuntu1-0-12.domain.local:9092 NodeId 2 [thrd:main]: ubuntu1-0-12.domain.local:9092/2: Enabled low-latency ops queue wake-ups [thrd:main]: ubuntu1-0-12.domain.local:9092/2: Added new broker with NodeId 2 [thrd:main]: ubunutu1-0-13.domain.local:9092/bootstrap: Broker #1/3: ubuntu1-0-13.domain.local:9092 NodeId 3 [thrd:main]: ubuntu1-0-13.domain.local:9092/3: Enabled low-latency ops queue wake-ups [thrd:main]: ubuntu1-0-13.domain.local:9092/3: Added new broker with NodeId 3 [thrd:main]: ubunutu1-0-13.domain.local:9092/bootstrap: Broker #2/3: ubuntu1-0-11.domain.local:9092 NodeId 1 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Enter main broker thread [thrd:main]: ubuntu1-0-11.domain.local:9092/1: Enabled low-latency ops queue wake-ups [thrd:ubuntu1-0-13.domain.local:9092/3]: ubuntu1-0-13.domain.local:9092/3: Enter main broker thread [thrd:main]: ubuntu1-0-11.domain.local:9092/1: Added new broker with NodeId 1 [thrd:main]: ubunutu1-0-13.domain.local:9092/bootstrap: ClusterId update "" -> "xD4GZxzuQ1mA0xU_4HFxCA" [thrd:main]: ubunutu1-0-13.domain.local:9092/bootstrap: ControllerId update -1 -> 3 [thrd:main]: Broadcasting state change [thrd:ubuntu1-0-11.domain.local:9092/1]: ubuntu1-0-11.domain.local:9092/1: Enter main broker thread [thrd:app]: New local topic: ProduceStateEntriesAsync [thrd:app]: NEW ProduceStateEntriesAsync [-1] 0000021FA309D590 (at rd_kafka_topic_new0:393) [thrd:main]: Topic ProduceStateEntriesAsync metadata information unknown [thrd:main]: Topic ProduceStateEntriesAsync partition count is zero: should refresh metadata [thrd:main]: Requesting metadata for 1/1 topics: refresh unavailable topics [thrd:main]: ubunutu1-0-13.domain.local:9092/bootstrap: Request metadata for 1 topic(s): refresh unavailable topics [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Sent MetadataRequest (v2, 51 bytes @ 0, CorrId 3) [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Received MetadataResponse (v2, 242 bytes, CorrId 3, rtt 19.32ms) [thrd:main]: ubunutu1-0-13.domain.local:9092/bootstrap: ===== Received metadata (for 1 requested topics): refresh unavailable topics ===== [thrd:main]: ubunutu1-0-13.domain.local:9092/bootstrap: ClusterId: xD4GZxzuQ1mA0xU_4HFxCA, ControllerId: 3 [thrd:main]: ubunutu1-0-13.domain.local:9092/bootstrap: 3 brokers, 1 topics [thrd:main]: ubunutu1-0-13.domain.local:9092/bootstrap: Broker #0/3: ubuntu1-0-12.domain.local:9092 NodeId 2 [thrd:main]: ubunutu1-0-13.domain.local:9092/bootstrap: Broker #1/3: ubuntu1-0-13.domain.local:9092 NodeId 3 [thrd:main]: ubunutu1-0-13.domain.local:9092/bootstrap: Broker #2/3: ubuntu1-0-11.domain.local:9092 NodeId 1 [thrd:main]: ubunutu1-0-13.domain.local:9092/bootstrap: Topic #0/1: ProduceStateEntriesAsync with 1 partitions [thrd:main]: Topic ProduceStateEntriesAsync changed state unknown -> exists [thrd:main]: Topic ProduceStateEntriesAsync partition count changed from 0 to 1 [thrd:main]: NEW ProduceStateEntriesAsync [0] 0000021FA309D000 (at rd_kafka_topic_partition_cnt_update:620) [thrd:main]: Topic ProduceStateEntriesAsync partition 0 Leader 2 [thrd:main]: ProduceStateEntriesAsync [0]: delegate to broker ubuntu1-0-12.domain.local:9092/2 (rktp 0000021FA309D000, term 0, ref 2, remove 0) [thrd:main]: ProduceStateEntriesAsync [0]: broker ubuntu1-0-12.domain.local:9092/2 is now leader for partition with 0 messages (0 bytes) queued [thrd:main]: Migrating topic ProduceStateEntriesAsync [0] 0000021FA309D000 from (none) to ubuntu1-0-12.domain.local:9092/2 (sending PARTITION_JOIN to ubuntu1-0-12.domain.local:9092/2) [thrd:main]: Partitioning 1 unassigned messages in topic ProduceStateEntriesAsync to 1 partitions [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Topic ProduceStateEntriesAsync [0]: joining broker (rktp 0000021FA309D000, 0 message(s) queued) [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Added ProduceStateEntriesAsync [0] to active list (1 entries, opv 0, 0 messages queued) [thrd:ubuntu1-0-12.domain.local:9092/2]: Broadcasting state change [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ProduceStateEntriesAsync [0] 1 message(s) in xmit queue (1 added from partition queue) [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ProduceStateEntriesAsync [0] 1 message(s) queued but broker not up [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Broker changed state INIT -> TRY_CONNECT [thrd:ubuntu1-0-12.domain.local:9092/2]: Broadcasting state change [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: broker in state TRY_CONNECT connecting [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Broker changed state TRY_CONNECT -> CONNECT [thrd:ubuntu1-0-12.domain.local:9092/2]: Broadcasting state change [thrd:main]: 1/1 messages were partitioned in topic ProduceStateEntriesAsync [thrd:main]: ubunutu1-0-13.domain.local:9092/bootstrap: 1/1 requested topic(s) seen in metadata [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Connecting to ipv4#10.1.0.12:9092 (plaintext) with socket 4320 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ProduceStateEntriesAsync [0] 1 message(s) in xmit queue (0 added from partition queue) [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ProduceStateEntriesAsync [0] 1 message(s) queued but broker not up [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ProduceStateEntriesAsync [0] 1 message(s) in xmit queue (0 added from partition queue) [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ProduceStateEntriesAsync [0] 1 message(s) queued but broker not up [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Connected to ipv4#10.1.0.12:9092 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Connected (#1) [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Updated enabled protocol features +ApiVersion to ApiVersion [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Broker changed state CONNECT -> APIVERSION_QUERY [thrd:ubuntu1-0-12.domain.local:9092/2]: Broadcasting state change [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Sent ApiVersionRequest (v0, 25 bytes @ 0, CorrId 1) [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ProduceStateEntriesAsync [0] 1 message(s) in xmit queue (0 added from partition queue) [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ProduceStateEntriesAsync [0] 1 message(s) queued but broker not up [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Received ApiVersionResponse (v0, 264 bytes, CorrId 1, rtt 19.50ms) [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Broker API support: [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey Produce (0) Versions 0..7 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey Fetch (1) Versions 0..10 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey Offset (2) Versions 0..4 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey Metadata (3) Versions 0..7 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey LeaderAndIsr (4) Versions 0..1 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey StopReplica (5) Versions 0..0 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey UpdateMetadata (6) Versions 0..4 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey ControlledShutdown (7) Versions 0..1 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey OffsetCommit (8) Versions 0..6 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey OffsetFetch (9) Versions 0..5 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey GroupCoordinator (10) Versions 0..2 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey JoinGroup (11) Versions 0..3 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey Heartbeat (12) Versions 0..2 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey LeaveGroup (13) Versions 0..2 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey SyncGroup (14) Versions 0..2 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey DescribeGroups (15) Versions 0..2 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey ListGroups (16) Versions 0..2 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey SaslHandshake (17) Versions 0..1 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey ApiVersion (18) Versions 0..2 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey CreateTopics (19) Versions 0..3 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey DeleteTopics (20) Versions 0..3 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey DeleteRecords (21) Versions 0..1 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey InitProducerId (22) Versions 0..1 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey OffsetForLeaderEpoch (23) Versions 0..2 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey AddPartitionsToTxn (24) Versions 0..1 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey AddOffsetsToTxn (25) Versions 0..1 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey EndTxn (26) Versions 0..1 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey WriteTxnMarkers (27) Versions 0..0 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey TxnOffsetCommit (28) Versions 0..2 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey DescribeAcls (29) Versions 0..1 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey CreateAcls (30) Versions 0..1 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey DeleteAcls (31) Versions 0..1 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey DescribeConfigs (32) Versions 0..2 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey AlterConfigs (33) Versions 0..1 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey AlterReplicaLogDirs (34) Versions 0..1 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey DescribeLogDirs (35) Versions 0..1 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey SaslAuthenticate (36) Versions 0..0 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey CreatePartitions (37) Versions 0..1 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey CreateDelegationToken (38) Versions 0..1 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey RenewDelegationToken (39) Versions 0..1 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey ExpireDelegationToken (40) Versions 0..1 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey DescribeDelegationToken (41) Versions 0..1 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ApiKey DeleteGroups (42) Versions 0..1 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Feature MsgVer1: Produce (2..2) supported by broker [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Feature MsgVer1: Fetch (2..2) supported by broker [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Enabling feature MsgVer1 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Feature MsgVer2: Produce (3..3) supported by broker [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Feature MsgVer2: Fetch (4..4) supported by broker [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Enabling feature MsgVer2 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Feature ApiVersion: ApiVersion (0..0) supported by broker [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Enabling feature ApiVersion [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Feature BrokerGroupCoordinator: GroupCoordinator (0..0) supported by broker [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Enabling feature BrokerGroupCoordinator [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Feature BrokerBalancedConsumer: GroupCoordinator (0..0) supported by broker [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Feature BrokerBalancedConsumer: OffsetCommit (1..2) supported by broker [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Feature BrokerBalancedConsumer: OffsetFetch (1..1) supported by broker [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Feature BrokerBalancedConsumer: JoinGroup (0..0) supported by broker [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Feature BrokerBalancedConsumer: SyncGroup (0..0) supported by broker [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Feature BrokerBalancedConsumer: Heartbeat (0..0) supported by broker [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Feature BrokerBalancedConsumer: LeaveGroup (0..0) supported by broker [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Enabling feature BrokerBalancedConsumer [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Feature ThrottleTime: Produce (1..2) supported by broker [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Feature ThrottleTime: Fetch (1..2) supported by broker [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Enabling feature ThrottleTime [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Feature Sasl: JoinGroup (0..0) supported by broker [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Enabling feature Sasl [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Feature SaslHandshake: SaslHandshake (0..0) supported by broker [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Enabling feature SaslHandshake [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Feature LZ4: GroupCoordinator (0..0) supported by broker [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Enabling feature LZ4 [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Feature OffsetTime: Offset (1..1) supported by broker [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Enabling feature OffsetTime [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Feature IdempotentProducer: InitProducerId (0..0) supported by broker [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Enabling feature IdempotentProducer [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Feature ZSTD: Produce (7..7) supported by broker [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Feature ZSTD: Fetch (10..10) supported by broker [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Enabling feature ZSTD [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Updated enabled protocol features to MsgVer1,ApiVersion,BrokerBalancedConsumer,ThrottleTime,Sasl,SaslHandshake,BrokerGroupCoordinator,LZ4,OffsetTime,MsgVer2,IdempotentProducer,ZSTD [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Broker changed state APIVERSION_QUERY -> UP [thrd:ubuntu1-0-12.domain.local:9092/2]: Broadcasting state change [thrd:ubuntu1-0-12.domain.local:9092/2]: Skipping metadata refresh of 1 topic(s): connected: already being requested [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ProduceStateEntriesAsync [0] 1 message(s) in xmit queue (0 added from partition queue) [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ProduceStateEntriesAsync [0]: Produce MessageSet with 1 message(s) (169437 bytes, ApiVersion 7, MsgVersion 2, MsgId 0, BaseSeq -1, PID{Invalid}) [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Sent partial ProduceRequest (v7, 0+169507/169508 bytes, CorrId 2) [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Sent ProduceRequest (v7, 169508 bytes @ 169507, CorrId 2) [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Received ProduceResponse (v7, 68 bytes, CorrId 2, rtt 79.24ms) [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: ProduceStateEntriesAsync [0]: MessageSet with 1 message(s) (MsgId 0, BaseSeq -1) delivered [thrd:main]: Requesting metadata for 1/1 topics: periodic refresh [thrd:main]: ubunutu1-0-12.domain.local:9092/bootstrap: Request metadata for 1 topic(s): periodic refresh [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Sent MetadataRequest (v2, 51 bytes @ 0, CorrId 2) [thrd:ubunutu1-0-12.domain.local:9092/bootstrap]: ubunutu1-0-12.domain.local:9092/bootstrap: Received MetadataResponse (v2, 242 bytes, CorrId 2, rtt 24.44ms) [thrd:main]: ubunutu1-0-12.domain.local:9092/bootstrap: ===== Received metadata (for 1 requested topics): periodic refresh ===== [thrd:main]: ubunutu1-0-12.domain.local:9092/bootstrap: ClusterId: xD4GZxzuQ1mA0xU_4HFxCA, ControllerId: 3 [thrd:main]: ubunutu1-0-12.domain.local:9092/bootstrap: 3 brokers, 1 topics [thrd:main]: ubunutu1-0-12.domain.local:9092/bootstrap: Broker #0/3: ubuntu1-0-12.domain.local:9092 NodeId 2 [thrd:main]: ubunutu1-0-12.domain.local:9092/bootstrap: Broker #1/3: ubuntu1-0-13.domain.local:9092 NodeId 3 [thrd:main]: ubunutu1-0-12.domain.local:9092/bootstrap: Broker #2/3: ubuntu1-0-11.domain.local:9092 NodeId 1 [thrd:main]: ubunutu1-0-12.domain.local:9092/bootstrap: Topic #0/1: ProduceStateEntriesAsync with 1 partitions [thrd:main]: Topic ProduceStateEntriesAsync partition 0 Leader 2 [thrd:main]: ubunutu1-0-12.domain.local:9092/bootstrap: 1/1 requested topic(s) seen in metadata [thrd:main]: Requesting metadata for 1/1 topics: periodic refresh [thrd:main]: ubunutu1-0-13.domain.local:9092/bootstrap: Request metadata for 1 topic(s): periodic refresh [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Sent MetadataRequest (v2, 51 bytes @ 0, CorrId 4) [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Received MetadataResponse (v2, 242 bytes, CorrId 4, rtt 25.79ms) [thrd:main]: ubunutu1-0-13.domain.local:9092/bootstrap: ===== Received metadata (for 1 requested topics): periodic refresh ===== [thrd:main]: ubunutu1-0-13.domain.local:9092/bootstrap: ClusterId: xD4GZxzuQ1mA0xU_4HFxCA, ControllerId: 3 [thrd:main]: ubunutu1-0-13.domain.local:9092/bootstrap: 3 brokers, 1 topics [thrd:main]: ubunutu1-0-13.domain.local:9092/bootstrap: Broker #0/3: ubuntu1-0-12.domain.local:9092 NodeId 2 [thrd:main]: ubunutu1-0-13.domain.local:9092/bootstrap: Broker #1/3: ubuntu1-0-13.domain.local:9092 NodeId 3 [thrd:main]: ubunutu1-0-13.domain.local:9092/bootstrap: Broker #2/3: ubuntu1-0-11.domain.local:9092 NodeId 1 [thrd:main]: ubunutu1-0-13.domain.local:9092/bootstrap: Topic #0/1: ProduceStateEntriesAsync with 1 partitions [thrd:main]: Topic ProduceStateEntriesAsync partition 0 Leader 2 [thrd:main]: ubunutu1-0-13.domain.local:9092/bootstrap: 1/1 requested topic(s) seen in metadata [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: failed: err: Local: Broker transport failure: (errno: No error) [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Disconnected (after 600254ms in state UP) [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Broker changed state UP -> DOWN [thrd:ubuntu1-0-12.domain.local:9092/2]: Broadcasting state change [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Purging bufq with 0 buffers [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Purging bufq with 0 buffers [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Updating 0 buffers on connection reset [thrd:ubuntu1-0-12.domain.local:9092/2]: Requesting metadata for 1/1 topics: broker down [thrd:ubuntu1-0-12.domain.local:9092/2]: ubunutu1-0-13.domain.local:9092/bootstrap: Request metadata for 1 topic(s): broker down [thrd:ubuntu1-0-12.domain.local:9092/2]: ubuntu1-0-12.domain.local:9092/2: Broker changed state DOWN -> INIT [thrd:ubuntu1-0-12.domain.local:9092/2]: Broadcasting state change [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Sent MetadataRequest (v2, 51 bytes @ 0, CorrId 5) [thrd:ubunutu1-0-13.domain.local:9092/bootstrap]: ubunutu1-0-13.domain.local:9092/bootstrap: Received MetadataResponse (v2, 242 bytes, CorrId 5, rtt 11.55ms) [thrd:main]: ubunutu1-0-13.domain.local:9092/bootstrap: ===== Received metadata (for 1 requested topics): broker down ===== [thrd:main]: ubunutu1-0-13.domain.local:9092/bootstrap: ClusterId: xD4GZxzuQ1mA0xU_4HFxCA, ControllerId: 3 [thrd:main]: ubunutu1-0-13.domain.local:9092/bootstrap: 3 brokers, 1 topics [thrd:main]: ubunutu1-0-13.domain.local:9092/bootstrap: Broker #0/3: ubuntu1-0-12.domain.local:9092 NodeId 2 [thrd:main]: ubunutu1-0-13.domain.local:9092/bootstrap: Broker #1/3: ubuntu1-0-13.domain.local:9092 NodeId 3 [thrd:main]: ubunutu1-0-13.domain.local:9092/bootstrap: Broker #2/3: ubuntu1-0-11.domain.local:9092 NodeId 1 [thrd:main]: ubunutu1-0-13.domain.local:9092/bootstrap: Topic #0/1: ProduceStateEntriesAsync with 1 partitions [thrd:main]: Topic ProduceStateEntriesAsync partition 0 Leader 2 [thrd:main]: ubunutu1-0-13.domain.local:9092/bootstrap: 1/1 requested topic(s) seen in metadata

@mhowlett After some times test, it seems not the issue of broker, whether on Windows or Linux VM.

I found all versions (0.11.3-0.11.6, 1.0.0 beta3) of Confluent.Kafka library are works well in Windows Console application and Windows Service. But for my special use case in Microsoft Exchange Server Transport Agent, it's very strangely that only 0.11.3 & 1.0.0 beta3 works, 0.11.4/0.11.5/0.11.6 may have some break changes which result in my library does not work.

Here are the detail test result:

#	Confluent.Kafka Version	librdkafka Version	Broker Server & Address	Console App	Windows Service	Exchange Agent
1	1.0.0 beta3	1.0.0 RC7	Windows Server (10.0.0.6:9092)	OK	OK	OK
2	0.11.6	0.11.6	Windows Server (10.0.0.6:9092)	OK	OK	ERROR
3	0.11.4	0.11.4	Windows Server (10.0.0.6:9092)	OK	OK	ERROR
4	0.11.3	0.11.3	Windows Server (10.0.0.6:9092)	OK	OK	OK
5	0.11.3	0.11.4	Windows Server (10.0.0.6:9092)	OK	OK	OK

All the source code are here: https://github.com/Li-Yanzhi/kafkatest

@cmills2000 - can you paste your code (as complete as possible). Is it a simple console app?

@Li-Yanzhi - The async code had a deadlock issue in 0.11.4 and prior. In 0.11.6 this should have been fixed. The producer code has completely changed going to 1.0-beta3, but IIRC the way async methods are handled is the same (so it's a bit surprising there is an issue in 0.11.6 but not 1.0-beta3, but maybe i'm just forgetting something). At any rate, you should be using the 1.0 API going forward - We most likely won't prioritize looking into this issue for previous releases (unless a specific customer support case calls for it).

Hi @mhowlett,

The example code using a Console App seems to work fine producing:

var dr = await p.ProduceAsync("test-topic", new Message<Null, string> { Value = "test" });

My code uses generics and a custom serializer to produce like so:

    public async Task<DeliveryResult<TKey, TMessage>> ProduceAsync<TKey, TMessage>(string topic, TKey key, TMessage message)
    {
        Producer<TKey, TMessage> producer = GetProducer<TKey, TMessage>(async: true);
        DeliveryResult<TKey, TMessage> deliveryResult = null;

        try
        {
            deliveryResult = await producer.ProduceAsync(topic, new Message<TKey, TMessage> { Value = message, Key = key });
        }
        catch (Exception e)
        {
            deliveryResult = await producer.ProduceAsync(topic, new Message<TKey, TMessage> { Value = message, Key = key });
            Console.WriteLine(e);
        }

        return deliveryResult;
    }

My message serializer just converts the message to json and then into binary like so:

public async Task<byte[]> SerializeAsync(T data, bool isKey, MessageMetadata messageMetadata, TopicPartition destination) { string json = JsonConvert.SerializeObject(data, settings); byte[] bytes = await json.ToByteArrayAsync(); return bytes; }

@cmills2000 - can you paste a complete minimal program demonstrating the problem - without the full program i'm just guessing... (and I don't have any good ideas).

Hi @mhowlett,

So I did a minimal program as a console app, and it worked, so nothing to be gained there. My prod code is more involved and it was hard to cut it down, so I just decided to download the source code (fixmk branch) and reference that instead of the nuget package and, surprise, surprise, it works.

Then I switched back to the 1.0.0-beta3 nuget package and the same problem of not returning is there still.

So there may be something amiss with the nuget package in combination with my setup. What that issue is, I couldn't tell you. If you want to go deeper, we could arrange something for sure.

I finally found that the issue was caused by library of System.Runtime.CompilerService.Unsafe which was introduced from 0.11.4 as dependency of Confluent.Kafka.

If I use Confluent.Kafka 0.11.x, I have to use the v4.0.0 of System.Runtime.CompilerService.Unsafe, the higher versions will cause producer.ProduceAsync to hang and never returns.

If I use Confluent.Kafka 1.0.0 beta3, I must use System.Runtime.CompilerService.Unsafe v4.5.1 or lower version, the latest version of v4.5.2 will cause producer.ProduceAsync to hang and never returns.

Hope this can help people who have same issues.

@Li-Yanzhi - that is great information! we need to do something better than the current behavior.

@Li-Yanzhi thanks, I can confirm that downgrading to v4.5.1 of System.Runtime.CompilerService.Unsafe fixes my problem as well with the Confluent.Kafka 1.0.0 beta3 nuget package as I can now call await ProduceAsync(...) and the call returns.

thanks for confirming @cmills2000 - the power of open source at work! I plan to get this resolved before the next release.

the dependency on System.Runtime.CompilerService.Unsafe has been removed in the 1.0.x branch - this'll be in the next release. thanks again @Li-Yanzhi for hunting down the problem and @cmills2000 for testing.

Hi, I ran into the same issue with the latest version of the Confluent .NET Kafka library. Downgrading of System.Runtime.CompilerService.Unsafe does not work because Kafka Nuget Package requires at least version 4.5.2 of the System.Runtime.CompilerService.Unsafe Are there any workarounds? And no, I am not running Kafka on Windows.

Hi All - I'm using "Confluent.Kafka 1.1.0" and downgraded System.Runtime.CompilerService.Unsafe to 4.5.0 and 4.5.1. Both are not returning back from "ProduceAsync". Console application code getting terminated without any error as soon as run on this line. Please share any inputs. Thanks in advance.

There is no longer a dependency on System.Runtime.CompilerService.Unsafe, so specifying a different version of this won't make any difference. @arunmani87 - what do you mean getting terminated? are you awaiting the call or calling Result? please share your code and or debug logs for more input.

Thanks mhowlett for your response. Confirmed that It works with latest version. Fixed the issue by add wait() to async call.

I'm having the same issue using "Confluent.Kafka 1.1.0". My stack:

.NET Core SDK 2.1 Kafka 2.3.0 over a RHEL 6 VM

Any thoughts?

@douglaslimafranco - there's most likely either a problem with your Kafka setup (we can probably see what that is if you provide debug logs) or with how your usingProduceAsync (which we can identify if you provide code). we aren't aware of any current issue in the library related to this thread, but we'll investigate specific scenarios reported.

My server.properties file:

Licensed to the Apache Software Foundation (ASF) under one or more

contributor license agreements. See the NOTICE file distributed with

this work for additional information regarding copyright ownership.

The ASF licenses this file to You under the Apache License, Version 2.0

(the "License"); you may not use this file except in compliance with

the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software

distributed under the License is distributed on an "AS IS" BASIS,

WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

See the License for the specific language governing permissions and

limitations under the License.

see kafka.server.KafkaConfig for additional details and defaults

############################# Server Basics #############################

The id of the broker. This must be set to a unique integer for each broker.

broker.id=0

############################# Socket Server Settings #############################

The address the socket server listens on. It will get the value returned from

java.net.InetAddress.getCanonicalHostName() if not configured.

FORMAT:

listeners = listener_name://host_name:port

EXAMPLE:

listeners = PLAINTEXT://your.host.name:9092

listeners=PLAINTEXT://:9092

listeners=PLAINTEXT://172.17.134.98:9092,SSL://172.17.134.98:9093,SASL_SSL://172.17.134.98:9094

listeners=PLAINTEXT://:9092,SSL://:9093,SASL_SSL://:9094

Hostname and port the broker will advertise to producers and consumers. If not set,

it uses the value for "listeners" if configured. Otherwise, it will use the value

returned from java.net.InetAddress.getCanonicalHostName().

advertised.listeners=PLAINTEXT://your.host.name:9092

advertised.listeners=PLAINTEXT://172.17.134.98:9092,SSL://172.17.134.98:9093,SASL_SSL://172.17.134.98:9094

Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details

listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL

listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_SSL:SASL_SSL

The number of threads that the server uses for receiving requests from the network and sending responses to the network

num.network.threads=3

The number of threads that the server uses for processing requests, which may include disk I/O

num.io.threads=8

The send buffer (SO_SNDBUF) used by the socket server

socket.send.buffer.bytes=102400

The receive buffer (SO_RCVBUF) used by the socket server

socket.receive.buffer.bytes=102400

The maximum size of a request that the socket server will accept (protection against OOM)

socket.request.max.bytes=104857600 l

############################# Log Basics #############################

A comma separated list of directories under which to store log files

log.dirs=/tmp/kafka-logs

The default number of log partitions per topic. More partitions allow greater

parallelism for consumption, but this will also result in more files across

the brokers.

num.partitions=1

The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.

This value is recommended to be increased for installations with data dirs located in RAID array.

num.recovery.threads.per.data.dir=1

############################# Internal Topic Settings #############################

The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"

For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3.

offsets.topic.replication.factor=1 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1

############################# Log Flush Policy #############################

Messages are immediately written to the filesystem but by default we only fsync() to sync

the OS cache lazily. The following configurations control the flush of data to disk.

There are a few important trade-offs here:

1. Durability: Unflushed data may be lost if you are not using replication.

2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.

3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to excessive seeks.

The settings below allow one to configure the flush policy to flush data after a period of time or

every N messages (or both). This can be done globally and overridden on a per-topic basis.

The number of messages to accept before forcing a flush of data to disk

log.flush.interval.messages=10000

The maximum amount of time a message can sit in a log before we force a flush

log.flush.interval.ms=1000

############################# Log Retention Policy #############################

The following configurations control the disposal of log segments. The policy can

be set to delete segments after a period of time, or after a given size has accumulated.

A segment will be deleted whenever either of these criteria are met. Deletion always happens

from the end of the log.

The minimum age of a log file to be eligible for deletion due to age

log.retention.hours=168

A size-based retention policy for logs. Segments are pruned from the log unless the remaining

segments drop below log.retention.bytes. Functions independently of log.retention.hours.

log.retention.bytes=1073741824

The maximum size of a log segment file. When this size is reached a new log segment will be created.

log.segment.bytes=1073741824

The interval at which log segments are checked to see if they can be deleted according

to the retention policies

log.retention.check.interval.ms=300000

############################# Zookeeper #############################

Zookeeper connection string (see zookeeper docs for details).

This is a comma separated host:port pairs, each corresponding to a zk

server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".

You can also append an optional chroot string to the urls to specify the

root directory for all kafka znodes.

zookeeper.connect=localhost:2181

Timeout in ms for connecting to zookeeper

zookeeper.connection.timeout.ms=6000

############################# Group Coordinator Settings #############################

The following configuration specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance.

The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms.

The default value for this is 3 seconds.

We override this to 0 here as it makes for a better out-of-the-box experience for development and testing.

However, in production environments the default value of 3 seconds is more suitable as this will help to avoid unnecessary, and potentially expensive, rebalances during application startup.

group.initial.rebalance.delay.ms=0

############################# SSL #############################

ssl.client.auth=none ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1

Server host name verification disabled

ssl.endpoint.identification.algorithm= # ssl.key.password=password ssl.keystore.location=/usr/pki/server.keystore.jks ssl.keystore.password=password ssl.keystore.type=JKS ssl.truststore.location=/usr/pki/server.truststore.jks ssl.truststore.password=password ssl.truststore.type=JKS

############################# JAAS #############################

listener.name.sasl_ssl.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \ username="admin" \ password="admin-secret" \ user_admin="admin-secret" \ user_kafka="kafka-secret";

############################# SASL #############################

security.inter.broker.protocol=SASL_SSL sasl.mechanism.inter.broker.protocol=PLAIN sasl.enabled.mechanisms=PLAIN

############################# Authorization and ACLs #############################

authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer allow.everyone.if.no.acl.found=true super.users=User:admin;User:kafka

My code:

internal async Task SendMessageAsync(MessageSent messageNotSent, DbSession dbSession) { var timestamp = DateTime.Now;

        var brokerMessage = CreateBrokerMessage(messageNotSent, timestamp);

        using (var producer = new ProducerBuilder<string, string>(_producerConfig).Build())
        {
            // Publish message to the broker.
            var deliveryReport = await producer.ProduceAsync(messageNotSent.Topic, brokerMessage);
        }

        UpdateMessageSent(messageNotSent, dbSession, timestamp);
    }

Which debug logs do you need?

I'm trying to connect over SSL in port 9093.

creating a producer is very expensive, you shouldn't do that for each message.

set the Debug config property to all (and set a log handler to get the messages if you can't read stderr).

I'm assuming my "stderr" would be my debug output window in Visual Studio. Is that correct?

If so, setting _clientConfig.Debug = "all"; doesn't seem to have made any difference, cause no logging entries from the library are popping up.

Unlike "Produce", "ProduceAsync" has no signature to receive a handler. How should I do that in this case.

set the log handler in the producer builder

Do you know in which circunstances I would have the following error?

|BROKERFAIL|rdkafka#producer-1| [thrd:sasl_ssl://172.17.134.98:9094/bootstrap]: sasl_ssl://172.17.134.98:9094/bootstrap: failed: err: Local: Broker transport failure: (errno: Unknown error)

I had a SASL_SSL misconfiguration. Now it's working. Thanks for your help.

I had the same error "await producer async never return". The cause was SSL wrong configuration. My recomendation: set the Debug config property to "all" and see the log

confluentinc / confluent-kafka-dotnet

await producer.ProduceAsync() never return #796