Kafka Trigger: Failed to load the librdkafka native library on macOS ARM64 (Apple Silicon M1/M2/M3)

[gastonmuijtjens]

When running an out-of-process isolated Azure Function v4 in .NET 6.0 using a Kafka Trigger (Microsoft.Azure.Functions.Worker.Extensions.Kafka) on macOS ARM64 (M1/M2/M3) the following error is encountered on startup:

[2023-11-28T15:23:44.201Z] A host error has occurred during startup operation '1f1ffdaf-dbc2-4c97-97a2-3bfc45e23ba0'.
[2023-11-28T15:23:44.202Z] Confluent.Kafka: Failed to load the librdkafka native library.
[2023-11-28T15:23:44.218Z] Failed to stop host instance '20345bf6-ed6a-476f-b069-334242f9eb8a'.
[2023-11-28T15:23:44.218Z] Microsoft.Azure.WebJobs.Host: The host has not yet started.

It looks like the native librdkafka binaries are not copied to the build output folder (i.e. bin/Debug/net6.0) on macOS ARM64.

To workaround this issue:

• Install librdkafka manually, for instance with Homebrew: brew install librdkafka
• Build the Azure Function Project
• Manually copy the following files in /opt/homebrew/Cellar/librdkafka/<version_number>/lib to the build output folder of the Function project (bin/Debug/net6.0):
  • librdkafka-static.a
  • librdkafka.1.dylib
  • librdkafka.a
  • librdkafka.dylib
  • librdkafka++.1.dylib
  • librdkafka++.a
  • librdkafka++.dylib
• Run the Azure Function Project

Note that you have to follow the above steps again if you rebuild the Function project.

See also

• https://github.com/Azure/azure-functions-kafka-extension/issues/425

OS Information

• macOS 14.1.1 ARM64
• .NET 6.0
• Microsoft.Azure.Functions.Worker.Extensions.Kafka 3.10.0
• Azure Functions Version V4 (dotnet-isolated)
• Azure Functions Core Tools: 4.0.5455

[kshyju]

@jainharsh98 Can you please take a look?

[kshyju]

@jainharsh98 / @lpapudippu Can you help with this?

[raman-nbg]

Any update on this on how to solve the issue?

[yatw]

This has been a problem for so long. To work around it, everyone in my team installed librdkafka from brew into /usr/local/Cellar/librdkafka Then copy the files over to /usr/local/lib where the Azure function is looking for these files

Now why is this librdkafka not included in the dependency for KafkaTrigger? Also the set up is confusing, there is the extensionBundle, there is the azure function core tools plugin, and the maven plugin, hard to tell what does what and which is responsible to load librdkafka