Open leiwang008 opened 2 months ago
Hi, I just got it done. I built the librdkafka into a static library and also got the dependencies as static library. Then everything works amazingly. By the way, I also upgrade to use the latest modern-cpp-kafka for this EasyKafka Plugin. If you like I can contribute my code to the project :-).
Hey @leiwang008 Three rules of thumb :B 1) use static libraries with UE, it will save you some tears. 2) If you want to support linux, make sure to use libc++ and not libstdc++ (librdkafka is c library as far as i can remember so doesn't matter). 3) Always compile against ur UE thirdparty libraries (Like OpenSSL, ZLib) - If you are using dynamic linked library its a must.
For Windows VCPKG is really handy For Linux you can use the cross compile toolchain from EPIC (clang), or even compiling on native Linux again using CLANG (Remember Libc++ :) ).
Hi @sha3sha3 Thank you very much for the detail explaination 👍 We just need to get it run on the Windows for now, no need for Linux.
I did run into the conflict problem for the (OpenSSL, ZLib), I fixed it by commenting out the UE library and use the static libraries (built from librdkafka) instead, do you think this is a problem.
string LibPath = Path.Combine(ModuleDirectory, "lib/Win64");
PublicAdditionalLibraries.Add(Path.Combine(LibPath, "lz4.lib"));
PublicAdditionalLibraries.Add(Path.Combine(LibPath, "rdkafka.lib"));
PublicAdditionalLibraries.Add(Path.Combine(LibPath, "rdkafka++.lib"));
PublicAdditionalLibraries.Add(Path.Combine(LibPath, "libcrypto.lib"));
PublicAdditionalLibraries.Add(Path.Combine(LibPath, "libcurl.lib"));
PublicAdditionalLibraries.Add(Path.Combine(LibPath, "libssl.lib"));
PublicAdditionalLibraries.Add(Path.Combine(LibPath, "zlib.lib"));
PublicAdditionalLibraries.Add(Path.Combine(LibPath, "zstd.lib"));
// comment out the following, there are conflicts between libcrypto.lib/libssl.lib(Version: 3.0.8)
// and UE5'S OpenSSL(C:\Program Files\Epic Games\UE_5.3\Engine\Binaries\ThirdParty\OpenSSL)
/* PublicDependencyModuleNames.AddRange(
new string[]
{
"OpenSSL",
"zlib"
}
);*/
Hey @leiwang008
The problem is that you didn't compile Librdkafka with UE dependencies.
Your solution isn't practical. The moment you use any module dependent on OpenSSL you will face the same issue.
Unreal engine uses OpenSSL 1.1x, instead you compiled the Library with OpenSSL 3.x.
Hi @sha3sha3 Thanks for the valuable information, I used to be a Java developer and I am new to c/c++ and Unreal Engine. How can I build the librdkafka with UE dependencies? I am using Visual Studio 2022 to build the librdkafka. There is too few information related to UE on the google. Are UE dependencies in the folder C:\Program Files\Epic Games\UE_5.3\Engine\Binaries\? I found there is a folder C:\Program Files\Epic Games\UE_5.3\Engine\Binaries\ThirdParty\OpenSSL\Win64\VS2015 holding 2 .dll files libeay32.dll and ssleay32.dll? Are they the UE dependencies that I should use? When I built the librdkafka, it seems that the visual studio will download automatically the dependencies into the folder "... librdkafka\vcpkg_installed\x64-windows-static\x64-windows-static\lib", if I build the librdkafka against the UE denpendencies, which dependencies it will use? the UE denpendencies or the dependencies in "... librdkafka\vcpkg_installed\x64-windows-static\x64-windows-static\lib"?
Ok, I found the UE libraries C:\Program Files\Epic Games\UE_5.3\Engine\Source\ThirdParty\OpenSSL\1.1.1t C:\Program Files\Epic Games\UE_5.3\Engine\Source\ThirdParty\libcurl\8.4.0 C:\Program Files\Epic Games\UE_5.3\Engine\Source\ThirdParty\zlib\1.2.13
When I built librdkafka, it will downloaded automatically the libraries of following version openssl: 3.0.8 libcurl: 7.86.0-DEV zlib: 1.2.13
The libraries versions are not consistent except the "zlib".
I tried to build the librdkafka with EU provided libraries (OpenSSL\1.1.1t, ibcurl\8.4.0, zlib\1.2.13), I used the visual studio and I did this by add the "include" folders to "Include Directories" in the VC++ Directories section C:\Program Files\Epic Games\UE_5.3\Engine\Source\ThirdParty\zlib\1.2.13\include C:\Program Files\Epic Games\UE_5.3\Engine\Source\ThirdParty\OpenSSL\1.1.1t\include\Win64\VS2015 C:\Program Files\Epic Games\UE_5.3\Engine\Source\ThirdParty\libcurl\8.4.0\include
I also add the library folders to "Library Directories" in the VC++ Directories section C:\Program Files\Epic Games\UE_5.3\Engine\Source\ThirdParty\libcurl\8.4.0\lib\Win64\Release C:\Program Files\Epic Games\UE_5.3\Engine\Source\ThirdParty\OpenSSL\1.1.1t\lib\Win64\VS2015\Release C:\Program Files\Epic Games\UE_5.3\Engine\Source\ThirdParty\zlib\1.2.13\lib\Win64\Release
I could build it successfully and I got the librdkafka.lib and librdkafkacpp.lib, I copied them to the "UE-EasyKafka\Source\ThirdParty\KafkaLib\lib\Win64" and modified the KafkaLib.Build.cs to include them and use the UE libraries as below
string LibPath = Path.Combine(ModuleDirectory, "lib/Win64");
PublicAdditionalLibraries.Add(Path.Combine(LibPath, "lz4.lib"));
PublicAdditionalLibraries.Add(Path.Combine(LibPath, "librdkafka.lib"));
PublicAdditionalLibraries.Add(Path.Combine(LibPath, "librdkafkacpp.lib"));
PublicAdditionalLibraries.Add(Path.Combine(LibPath, "zstd.lib"));
PublicDependencyModuleNames.AddRange(
new string[]
{
"OpenSSL",
"zlib",
"libcurl"
}
);
But it gave me a lot of errors
Severity Code Description Project File Line Suppression State Details
Error LNK2019 unresolved external symbol imp_curl_easy_cleanup referenced in function rd_http_get connector C:\repository\github\connector\Intermediate\ProjectFiles\librdkafka.lib(rdhttp.obj) 1
Error LNK2019 unresolved external symbol __imp_curl_easy_cleanup referenced in function rd_http_get connector C:\repository\github\connector\Intermediate\ProjectFiles\librdkafka.lib(rdhttp.obj) 1
Error LNK2019 unresolved external symbol imp_curl_easy_cleanup referenced in function rd_http_get connector C:\repository\github\connector\Intermediate\ProjectFiles\librdkafka.lib(rdhttp.obj) 1
Error LNK2019 unresolved external symbol __imp_curl_easy_cleanup referenced in function rd_http_get connector C:\repository\github\connector\Intermediate\ProjectFiles\librdkafka.lib(rdhttp.obj) 1
I noticed that the library size are different between the UE's and automatically downloaded version, for example libcurl.lib, UE's version is 2180KB, but automatically downloaded version is much bigger, it is 12531KB
Do you have any idea about these errors, any suggestions?
Hi shasha, I know this project is not maintained, but you are the founder of this project and have many knowledge and experiences :-) . I would like to know if you have any idea about the problem I met, I appreciate any help.
I would like to use the "oauthbearer/oidc" feature provided by the librdkafka2.4.0 in our Unreal Project on platform Windows x86_64. I cloned your project and then I upgraded EasyKafka to use the https://github.com/morganstanley/modern-cpp-kafka/releases/tag/v2024.07.03 and I could get it built successfully. Then I tried to upgrade the librdkafka to use the version 2.4.0, the following is what I did
So I checked the "librdkafka" solution, I found that it depends some installed libraries which have been put in C:\sdk\librdkafka-2.4.0\vcpkg_installed\x64-windows\x64-windows\lib (libcurl.lib, libcrypto.lib, libssl.lib, zlib.lib,zstd.lib etc.) and copied them to the "UE-EasyKafka\Source\ThirdParty\KafkaLib\lib\Win64" and modified the KafkaLib.Build.cs to include them
but this no luck, cannot load the EasyKafka plugin.