Quickstream commented 5 years ago

Description

Getting the following exception when building a consumer using 1.0.0-beta3:

Unhandled Exception: System.DllNotFoundException: Failed to load the librdkafka native library. at Confluent.Kafka.Impl.Librdkafka.Initialize(String userSpecifiedPath) at Confluent.Kafka.Consumer2..ctor(ConsumerBuilder2 builder) at Confluent.Kafka.ConsumerBuilder2.Build() at Confluent.Kafka.Examples.ConsumerExample.Program.Run_Consume(String brokerList, List1 topics, CancellationToken cancellationToken) in /src/confluent-kafka-dotnet/examples/Consumer/Program.cs:line 60 at Confluent.Kafka.Examples.ConsumerExample.Program.Main(String[] args) in /src/confluent-kafka-dotnet/examples/Consumer/Program.cs:line 192 Abort trap: 6

How to reproduce

Run the Consumer example console app using 1.0.0-beta3. (https://github.com/confluentinc/confluent-kafka-dotnet/tree/master/examples/Consumer)

If I run the Consumer example app with from the 1.0.0-beta2 release, it works without error.

Checklist

Please provide the following information:

[x] Confluent.Kafka nuget version: 1.0.0-beta3
[ ] Apache Kafka version:
[ ] Client configuration:
[x] Operating system: OSX 10.13.6 (17G65)
[ ] Provide logs (with "debug" : "..." as necessary in configuration)
[ ] Provide broker log excerpts
[ ] Critical issue

mhowlett commented 5 years ago

thanks for reporting. this is because the zstd dependency of librdkafka 1.0-RC7 is not available on macos by default. for now you could try installing that (i have not, don't know what the process is, but guess it's not difficult), or compiling librdkafka / installing from source (this is easy). if you do the latter, the librdkafka version you have on your system will be automatically used when the version in the nuget package fails to load.

this issue will be fixed before the 1.0 final release.

edenhill commented 5 years ago

The workaround for now is brew install zstd on osx

brightermatter commented 5 years ago

We also had to upgrade OpenSSL on osx by brew upgrade openssl to resolve the issue

ritasker commented 5 years ago

I am also seeing this with 0.11.6 on linux. I have installed zstd as suggested in the workaround, this didn't fix the issue.

$ dotnet run -p src/Kafka.Consumer                                                                             

Unhandled Exception: System.DllNotFoundException: Failed to load the librdkafka native library.
   at Confluent.Kafka.Impl.LibRdKafka.Initialize(String userSpecifiedPath)
   at Confluent.Kafka.Consumer..ctor(IEnumerable`1 config)
   at Confluent.Kafka.Consumer`2..ctor(IEnumerable`1 config, IDeserializer`1 keyDeserializer, IDeserializer`1 valueDeserializer)
   at Kafka.Consumer.Program.Main(String[] args) in /home/ritasker/projects/Kafka/src/Kafka.Consumer/Program.cs:line 23

$ dotnet --info                                                                                                                            
.NET Core SDK (reflecting any global.json):
 Version:   2.2.102
 Commit:    96ff75a873

Runtime Environment:
 OS Name:     manjaro
 OS Version:  
 OS Platform: Linux
 RID:         arch-x64
 Base Path:   /opt/dotnet/sdk/2.2.102/

Host (useful for support):
  Version: 2.2.1
  Commit:  878dd11e62

.NET Core SDKs installed:
  2.2.102 [/opt/dotnet/sdk]

.NET Core runtimes installed:
  Microsoft.NETCore.App 2.2.1 [/opt/dotnet/shared/Microsoft.NETCore.App]

mhowlett commented 5 years ago

zstd is new in 1.0 - doesn't apply to 0.11.6. your problem will be that another required dependency of librdkafka is not on your system. you can use ldd for more info. Something like: ldd ~/.nuget/packages/librdkafka.redist/0.11.6/runtimes/linux-x64/native/librdkafka.so. This info might also help: https://github.com/edenhill/librdkafka/wiki/Dependencies

ritasker commented 5 years ago

Looks like I am missing libsasl2.so.2.

$ ldd ~/.nuget/packages/librdkafka.redist/0.11.6/runtimes/linux-x64/native/librdkafka.so
    linux-vdso.so.1 (0x00007ffe406e7000)
    libsasl2.so.2 => not found
    libssl.so.1.0.0 => /usr/lib/libssl.so.1.0.0 (0x00007f265f408000)
    libcrypto.so.1.0.0 => /usr/lib/libcrypto.so.1.0.0 (0x00007f265f19b000)
    libm.so.6 => /usr/lib/libm.so.6 (0x00007f265f016000)
    libz.so.1 => /usr/lib/libz.so.1 (0x00007f265edff000)
    libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f265edfa000)
    libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f265edd7000)
    librt.so.1 => /usr/lib/librt.so.1 (0x00007f265edcd000)
    libc.so.6 => /usr/lib/libc.so.6 (0x00007f265ec09000)
    /usr/lib64/ld-linux-x86-64.so.2 (0x00007f265f7ab000)

mhowlett commented 5 years ago

if you install that, it should work.

vertexmachina commented 5 years ago

@ritasker

Looks like you're on an Arch Linux derivative. With libsasl installed, the library is actually libsasl2.so.3 while the prebuilt librdkafka.so is attempting to link to libsasl2.so.2.

I manually built librdkafka myself so it would link against the right libsasl, and then replaced ~/.nuget/packages/librdkafka.redist/1.0.0-rc7/runtimes/linux-x64/native/librdkafka.so with the version I manually built.

That should fix it.

ritasker commented 5 years ago

Thanks @vertexmachina I will give it ago when I am back off holiday.

blackie1019 commented 5 years ago

@brightermatter You are right! This library need openssl. Once I install openssl to my macOS, issue was fixed! Thank you so much!

mhowlett commented 5 years ago

just wanted to note in this thread that the latest librdkafka.redist package (1.0-RC8) will work out of the box with many more (the majority of all) linux distros. it contains 4 builds for linux, which it will attempt to load in the following order:

librdkafka.so # ubuntu, depends on libsasl2, openssl 1.0.x
centos7-librdkafka.so # centos, depends on libsasl2, openssl 1.0.x (but different soname from ubuntu image)
debian9-librdkafka.so # debian, no dependencies, no kerberos support
alpine-librdkafka.so # alpine, no dependencies. no kerberos support

iaq3 commented 4 years ago

Hello,

I am running 1.2.0 on Unity 3D I have a basic consumer example, and am getting:

DllNotFoundException: kernel32
Confluent.Kafka.Impl.Librdkafka.Initialize (System.String userSpecifiedPath) (at <8f0e45c156f948c58d05e6d1b69d0ee8>:0)
Confluent.Kafka.Consumer`2[TKey,TValue]..ctor (Confluent.Kafka.ConsumerBuilder`2[TKey,TValue] builder) (at <8f0e45c156f948c58d05e6d1b69d0ee8>:0)

I have brew installed openssl and zstd. Are there any other suggestions?

Thank you for any help or pointing me to a better place to ask this question.

ratri-git commented 4 years ago

Use this in dockerfile , it was installed and worked fine.

FROM mcr.microsoft.com/dotnet/core/runtime:3.0 AS runtime
RUN apt-get update -y
RUN apt-get install -y librdkafka-dev

If you have any errors i will share my full dockerfile

szalapski commented 4 years ago

@ratri-git I added those lines but I still get "Failed to load the librdkafka native library."

Do you expect resulting files for librdkafka-dev to exist somewhere after running that apt-get?

Yes, I'd like to see the full dockerfile.

PaskeS commented 3 years ago

Following this just leads me to this issue -> https://github.com/confluentinc/confluent-kafka-dotnet/issues/1596

swimhiking commented 2 years ago

I have same issue after Ubuntu released 20.04.4; and Dockerfile cannot specify minor version.

amarj27 commented 2 years ago

I am having the same issue as in thread #1596 after following this. Any suggestions on how to fix this? I am using a NodeJS Azure Function which uses Extension Bundles.

swimhiking commented 2 years ago

I simply resolve this issue by install this lib in base image, so just modify your docker file, it should be good

amarj27 commented 2 years ago

@swimhiking Could you tell me exaactly what you did in your Dockerfile? This is what I have

FROM mcr.microsoft.com/azure-functions/node:3.0-node10-core-tools As core-tools

COPY . /home/site/wwwroot

RUN cd /home/site/wwwroot && \ func extensions install --javascript

To enable ssh & remote debugging on app service change the base image to the one below

FROM mcr.microsoft.com/azure-functions/node:3.0-appservice

FROM mcr.microsoft.com/azure-functions/node:3.0 WORKDIR /home/site/wwwroot

ENV AzureWebJobsScriptRoot=/home/site/wwwroot \ AzureFunctionsJobHostLoggingConsole__IsEnabled=true \ NODE_TLS_REJECT_UNAUTHORIZED=0 \ RUNTIME_VERSION=v2.3.1 \ NODE_OPTIONS="--max-old-space-size=24576" \ LD_LIBRARY_PATH=/home/site/wwwroot/bin/runtimes/linux-x64/native

COPY . /home/site/wwwroot

RUN apt-get update -y RUN apt-get install -y librdkafka-dev RUN apt-get install -y libc-dev

and I end up with this error - Invalid librdkafka version b06ff, expected at least 10502ff

If I dont install the librdkafka-dev library, then I get a Dll not found exception

swimhiking commented 2 years ago

I have: FROM mcr.microsoft.com/azure-functions/node:4.0-node16-slim as base ENV LD_LIBRARY_PATH=/home/site/wwwroot/bin/runtimes/linux-x64/native RUN apt-get -y update \ && apt-get -y upgrade \ && apt-get install -y librdkafka-dev

so looks similar but with version4.0, I install lib in above base image

amarj27 commented 2 years ago

After I updated my base image to node:4.0 and removed the step to install the libc-dev module, it looks like I have been able to get past the dll error.

foman commented 1 year ago

It also happens on mcr.microsoft.com/dotnet/aspnet:6.0-alpine image.

System.DllNotFoundException: Failed to load the librdkafka native library.\n   at bool Confluent.Kafka.Impl.Librdkafka.TrySetDelegates(List<Type> nativeMethodCandidateTypes)\n   at void Confluent.Kafka.Impl.Librdkafka.LoadLinuxDelegates(string userSpecifiedPath)\n   at bool Confluent.Kafka.Impl.Librdkafka.Initialize(string userSpecifiedPath)

it can be fixed byapk add librdkafka-dev , but could it be solved in the kafka dotnet library?

swimhiking commented 1 year ago

Try to install the library on dotnet environment.

On Sun, Jan 15, 2023 at 11:45 PM Tim-Liang @.***> wrote:

It also happens on mcr.microsoft.com/dotnet/aspnet:6.0-alpine image. System.DllNotFoundException: Failed to load the librdkafka native library.\n at bool Confluent.Kafka.Impl.Librdkafka.TrySetDelegates(List nativeMethodCandidateTypes)\n at void Confluent.Kafka.Impl.Librdkafka.LoadLinuxDelegates(string userSpecifiedPath)\n at bool Confluent.Kafka.Impl.Librdkafka.Initialize(string userSpecifiedPath)

it can be fixed by apk add librdkafka-dev , but could it be solved in the kafka dotnet library?

— Reply to this email directly, view it on GitHub https://github.com/confluentinc/confluent-kafka-dotnet/issues/778#issuecomment-1383765600, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABOF4X6VVY5EVXIDN6RXVNDWSUKFHANCNFSM4GXAP7PA . You are receiving this because you were mentioned.Message ID: @.***>

anchitj commented 3 weeks ago

Is this still an issue?

confluentinc / confluent-kafka-dotnet

Exception "Failed to load the librdkafka native library" when building consumer using 1.0.0-beta3 #778

Description

How to reproduce

Checklist

To enable ssh & remote debugging on app service change the base image to the one below

FROM mcr.microsoft.com/azure-functions/node:3.0-appservice