librdkafka Problem on Windows 11 with Parallels on Mac M2 Processor

[ibrahimozgon]

Description

I moved to Mac M2 and installed Windows 11 on Parallels. I have several projects use confluent kafka 2.02 version. All projects works on x64 windows machines and docker but fails to load kafka libraries on M2.

I try to run a .net core 6 project and I get System.DllNotFoundException: 'Failed to load the librdkafka native library.'

I try to run .net framework 4.8 project and it gets

System.ComponentModel.Win32Exception: %1 is not a valid Win32 application

Error while loading librdkafka.dll or its dependencies from C:\...\bin\librdkafka\x64\librdkafka.dll. Check the directory exists, if not check your deployment process. You can also load the library and its dependencies by yourself before any call to Confluent.Kafka]
   Confluent.Kafka.Impl.Librdkafka.LoadNetFrameworkDelegates(String userSpecifiedPath) +588
   Confluent.Kafka.Impl.Librdkafka.Initialize(String userSpecifiedPath) +188
   Confluent.Kafka.Producer`2..ctor(ProducerBuilder`2 builder) +440
   Confluent.Kafka.ProducerBuilder`2.Build() +80

Strange thing is I have "librdkafka.dll" in that folder.

I tried to install libdirkafka library by vcpkg on windows and copied lib files to bin and project root folder but didn't help.

I installed all the waiting windows updates, it didn't help too.

I'm using Microsoft Visual Studio Community 2022 (ARM 64-bit) - Current version.

How to reproduce

Use Mac M2 machine and install windows 11 on Parallels.

Try to run this code:

var consumer = new ConsumerBuilder<Ignore, string>(new ConsumerConfig
            {
                BootstrapServers = ..,
                SaslUsername = ..,
                SaslPassword = ..,
                SaslMechanism = Plain,
                SecurityProtocol = SaslPlaintext,
                GroupId = ..,
                EnableAutoCommit = true,
                StatisticsIntervalMs = 5000,
                SocketTimeoutMs = _5000,
                SessionTimeoutMs =  5000,
                AutoOffsetReset = AutoOffsetReset.Earliest,
                AllowAutoCreateTopics = true
            }).Build()

or

            var conf = new ProducerConfig
            {
                BootstrapServers = ..,
                SaslUsername = ..,
                SaslPassword = ..,
                SaslMechanism = Plain,
                SecurityProtocol = SaslPlaintext,
                SocketTimeoutMs = 5000,
                MessageTimeoutMs = 5000,
                EnableDeliveryReports = true
            };

            _producer = new ProducerBuilder<Null, T>(conf).Build();

Basic example to run:

using Confluent.Kafka;

Console.WriteLine("Hello, World!");

var conf = new ProducerConfig
{
    BootstrapServers = ..,
    SaslMechanism = SaslMechanism.Plain,
    SecurityProtocol = SecurityProtocol.Plaintext,
    SaslUsername = ..,
    SaslPassword = ..,
    EnableDeliveryReports = true
};
//fails here
var _producer = new ProducerBuilder<Null, string>(conf)
.Build();
//fails here.
_producer.Produce("test", new Message<Null, string> { Value = "test" });

Checklist

Please provide the following information:

• [ ] A complete (i.e. we can run it), minimal program demonstrating the problem. No need to supply a project file.
• [X] Confluent.Kafka nuget version.
• [ ] Apache Kafka version.
• [X] Client configuration.
• [X] Operating system. Windows 11 over Mac M2 processor
• [X] Provide logs (with "debug" : "..." as necessary in configuration).
• [ ] Provide broker log excerpts.
• [X] Critical issue.

[filipboshevski]

There's no runtime built for arm64 for Windows yet for some reason. I've created a discussion here, if anyone is facing this issue, please upvote it so we can get more attention to it faster. Thanks :D

[filipboshevski]

@anchitj Any update on this?

[alexeyzimarev]

Same happens on macOS ARM64 with .NET 8.