search

confluentinc / confluent-kafka-go

Confluent's Apache Kafka Golang client
Apache License 2.0
4.59k stars 652 forks source link

How to use Golang client on Windows #128

Closed abhirockzz closed 3 years ago

abhirockzz commented 6 years ago

Description

How can I configure windows in order for my Golang client to work ? e.g. where should I place librdkafka client ?

How to reproduce

  1. download librdkafka from https://www.nuget.org/packages/librdkafka.redist/ & unzip its contents
  2. go get -u github.com/confluentinc/confluent-kafka-go/kafka
  3. setup pkg-config for windows
  4. go run main.go results in
Package rdkafka was not found in the pkg-config search path.
Perhaps you should add the directory containing `rdkafka.pc'
to the PKG_CONFIG_PATH environment variable
No package 'rdkafka' found
pkg-config: exit status 1
edenhill commented 6 years ago

librdkafka.redist does not contain a pkg-config file (rdkafka.pc), you will need to construct one yourself. As a reference, this is where it is being generated for unix builds: https://github.com/edenhill/librdkafka/blob/0c2ed0a2289f926780d228698b30a71968292503/mklove/Makefile.base#L102

abhirockzz commented 6 years ago

Thanks. After having resolved the pkg-config file issue and setting up gcc, go run main.go gives me this

# github.com/confluentinc/confluent-kafka-go/kafka
..\..\go\src\github.com\confluentinc\confluent-kafka-go\kafka\00version.go:25:10: fatal error: librdkafka/rdkafka.h: No such file or directory
 #include <librdkafka/rdkafka.h>
          ^~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
edenhill commented 6 years ago

While librdkafka is widely used on Windows (especially with the .NET client), we have not attempted to use it with Go client or any other build tool than Visual Studio. It is not clear where the Go toolchain would look for libraries and include files on Windows, but that is probably documented somewhere in the Go docs.

If, when, you get this working, it would be very useful if you could write up what you had to do and where you had to place libraries, etc, to get it to build and run.

matthax commented 6 years ago

@edenhill I've embarked on this adventure, here's what I have so far.

  1. Install cygwin (You should be able to just install pkg-config as a standalone dependency too)
  2. Create the rdkafka.pc file: 
    "prefix=C:/`nlibdir=C:/cygwin64/lib/`nincludedir=C:/cygwin64/usr/include`n`nName: librdkafka`nDescription: The Apache Kafka C/C++ library`nVersion: 0.11.4`nCflags: -I`${includedir}`nLibs: -L`${libdir} -lrdkafka`nLibs.private: -lssl -lcrypto   -lcrypto   -lz   -ldl -lpthread -lrt" | Out-File C:/cygwin64/lib/pkgconfig/rdkafka.pc -Encoding UTF8

Output file:

prefix=C:/ libdir=C:/cygwin64/lib/ includedir=C:/cygwin64/usr/include Name: librdkafka Description: The Apache Kafka C/C++ library Version: 0.11.4 Cflags: -I${includedir} Libs: -L${libdir} -lrdkafka Libs.private: -lssl -lcrypto -lcrypto -lz -ldl -lpthread -lrt
  1. Install librdkafka: 
    nuget install librdkafka.redist -Version 0.11.4-RC1B
Copy-Item -Path .\librdkafka.redist.0.11.4-RC1B\build\native\include\librdkafka -Recurse -Destination C:\cygwin64\usr\include
$src = Join-Path (get-item env:"GOPATH").Value "src"
$app = Join-Path $src "my-go-project"
cd $app
go get -u github.com/confluentinc/confluent-kafka-go/kafka

I now get:

# github.com/confluentinc/confluent-kafka-go/kafka
/usr/lib/gcc/x86_64-pc-cygwin/6.4.0/../../../../x86_64-pc-cygwin/bin/ld: cannot find -lrdkafka
/usr/lib/gcc/x86_64-pc-cygwin/6.4.0/../../../../x86_64-pc-cygwin/bin/ld: cannot find -lrdkafka
collect2: error: ld returned 1 exit status

Previously got:

# github.com/confluentinc/confluent-kafka-go/kafka ..\github.com\confluentinc\confluent-kafka-go\kafka\generated_errors.go:106:36: could not determine kind of name for C.RD_KAFKA_RESP_ERR__NOENT ..\github.com\confluentinc\confluent-kafka-go\kafka\generated_errors.go:104:39: could not determine kind of name for C.RD_KAFKA_RESP_ERR__READ_ONLY ..\github.com\confluentinc\confluent-kafka-go\kafka\generated_errors.go:108:40: could not determine kind of name for C.RD_KAFKA_RESP_ERR__UNDERFLOW

Appears to be a linking failure, have to check if it's looking for .so files or what. Not sure...

edenhill commented 6 years ago

Nice!

The build errors are due to Go client master requires librdkafka master (or the upcoming 0.11.4 release). You can install the librdkafka.redist 0.11.4 release candidate: https://www.nuget.org/packages/librdkafka.redist/0.11.4-RC1B

matthax commented 6 years ago

Got past those, I've also resolved the linking issue. 

# github.com/confluentinc/confluent-kafka-go/kafka
C:\Users\mbark201\AppData\Local\Temp\go-build471690010\b001\_x007.o: In function `chdrs_to_tmphdrs':
..\github.com\confluentinc\confluent-kafka-go\kafka/event.go:38: undefined reference to `rd_kafka_header_get_all'
..\github.com\confluentinc\confluent-kafka-go\kafka/event.go:38:(.text+0x30): relocation truncated to fit: R_X86_64_PC32 against undefined symbol `rd_kafka_header_get_all'
C:\Users\mbark201\AppData\Local\Temp\go-build471690010\b001\_x007.o: In function `_rk_queue_poll':
..\github.com\confluentinc\confluent-kafka-go\kafka/event.go:67: undefined reference to `rd_kafka_message_headers'
..\github.com\confluentinc\confluent-kafka-go\kafka/event.go:67:(.text+0xb4): relocation truncated to fit: R_X86_64_PC32 against undefined symbol `rd_kafka_message_headers'
..\github.com\confluentinc\confluent-kafka-go\kafka/event.go:68: undefined reference to `rd_kafka_header_cnt'
..\github.com\confluentinc\confluent-kafka-go\kafka/event.go:68:(.text+0xe6): relocation truncated to fit: R_X86_64_PC32 against undefined symbol `rd_kafka_header_cnt'
C:\Users\mbark201\AppData\Local\Temp\go-build471690010\b001\_x016.o: In function `tmphdrs_to_chdrs':
..\github.com\confluentinc\confluent-kafka-go\kafka/producer.go:43: undefined reference to `rd_kafka_headers_new'
..\github.com\confluentinc\confluent-kafka-go\kafka/producer.go:43:(.text+0x19): relocation truncated to fit: R_X86_64_PC32 against undefined symbol `rd_kafka_headers_new'
..\github.com\confluentinc\confluent-kafka-go\kafka/producer.go:46: undefined reference to `rd_kafka_header_add'
..\github.com\confluentinc\confluent-kafka-go\kafka/producer.go:46:(.text+0x58): relocation truncated to fit: R_X86_64_PC32 against undefined symbol `rd_kafka_header_add'
collect2: error: ld returned 1 exit status

Wondering if I accidentally linked x32 and x64 somehow...

edenhill commented 6 years ago

I think it still picks up an older librdkafka.dll. Remove all old ones and only keep 0.11.4

matthax commented 6 years ago

Yep, works now. Had to copy librdkafka.so from runtimes\linux-x64\native into C:\cygwin64\lib

Copy-Item -Path .\librdkafka.redist.0.11.4-RC1B\runtimes\linux-x64\native\librdkafka.so -Destination C:\cygwin64\lib\librdkafka.so

and that seemed to do the trick...

edenhill commented 6 years ago

That's great news! First successful run of confluent-kafka-go on Windows 🥇

maustin75 commented 6 years ago

@matthax This is good news! Would you mind posting a step by step summary of how to get this to work? It would be greatly appreciated.

Thanks!

iradul commented 6 years ago

step-by-step that worked for me

Name: librdkafka Description: The Apache Kafka C/C++ library Version: 0.11.4 Cflags: -I${includedir} Libs: -L${libdir} -lrdkafka Libs.private: -lssl -lcrypto -lcrypto -lz -ldl -lpthread -lrt


* run `set CC=x86_64-w64-mingw32-gcc` (this will allow `cgo` to use `x86_64-w64-mingw32-gcc` instead of `gcc` - you can make sure it worked with `go env CC`)
* run `go build`, that will create executable
* make sure to deliver `librdkafka.dll`, `msvcr120.dll` and `zlib.dll` from `.\librdkafka.redist.0.11.4\runtimes\win7-x64\native\` alongside with newly created executable
yuanxiang250 commented 6 years ago

@iradul help help I'm on windows show error : go build github.com/confluentinc/confluent-kafka-go/kafka: C:\Go\pkg\tool\windows_amd64\cgo.exe: exit status 2

$ gcc -v Using built-in specs. COLLECT_GCC=C:\MinGW\bin\gcc.exe COLLECT_LTO_WRAPPER=c:/mingw/bin/../libexec/gcc/mingw32/6.3.0/lto-wrapper.exe Target: mingw32 Configured with: ../src/gcc-6.3.0/configure --build=x86_64-pc-linux-gnu --host=mingw32 --target=mingw32 --with-gmp=/mingw --with-mpfr --with-mpc=/mingw --with-isl=/mingw --prefix=/mingw --disable-win32-registry --with-arch=i586 --with-tune=generic --enable-languages=c,c++,objc,obj-c++,fortran,ada --with-pkgversion='MinGW.org GCC-6.3.0-1' --enable-static --enable-shared --enable-threads --with-dwarf2 --disable-sjlj-exceptions --enable-version-specific-runtime-libs --with-libiconv-prefix=/mingw --with-libintl-prefix=/mingw --enable-libstdcxx-debug --enable-libgomp --disable-libvtv --enable-nls Thread model: win32 gcc version 6.3.0 (MinGW.org GCC-6.3.0-1)

$ go version go version go1.9.4 windows/amd64

go get -x github.com/confluentinc/confluent-kafka-go/kafka WORK=C:\cygwin64\tmp\go-build923822198 mkdir -p $WORK\github.com\confluentinc\confluent-kafka-go\kafka_obj\ mkdir -p $WORK\github.com\confluentinc\confluent-kafka-go\ cd E:\gowork\GOPATH\src\github.com\confluentinc\confluent-kafka-go\kafka pkg-config --cflags -- rdkafka pkg-config --libs -- rdkafka CGO_LDFLAGS="-g" "-O2" "-Lc:/cygwin64/lib/" "-lrdkafka" "C:\Go\pkg\tool\windows_amd64\cgo.exe" -objdir "C:\cygwin64\tmp\go-build923822198\github.com\confluentinc\confluent-kafka-go\kafka\_obj\" -importpath github.com/confluentinc/confluent-kafka-go/kafka -- -Ic:/cygwin64/usr/include -I "C:\cygwin64\tmp\go-build923822198\github.com\confluentinc\confluent-kafka-go\kafka\_obj\" -g -O2 00version.go build_dynamic.go config.go consumer.go error.go event.go generated_errors.go handle.go header.go kafka.go message.go metadata.go misc.go offset.go producer.go testhelpers.go go build github.com/confluentinc/confluent-kafka-go/kafka: C:\Go\pkg\tool\windows_amd64\cgo.exe: exit status 2

iradul commented 6 years ago

@yuanxiang250 try to follow steps from above. You don't need to install MinGW, you just need x86_64-w64-mingw32-gcc that comes as a package for cygwin64 within installation. I uploaded installation step screenshot to clarify how to do it.

code-n-go commented 6 years ago

@iradul, thank you... very much!

CKFVL commented 6 years ago

I followed the steps @iradul mentioned and I get following error. please suggest.

Executing command line: [C:\Go\bin\go.exe, build, -o, build/app.go.bin, D:\projects\testkafka\src\datatest/main.go] go build github.com/confluentinc/confluent-kafka-go/kafka: build constraints exclude all Go files in D:\projects\testkafka\src\github.com\confluentinc\confluent-kafka-go\kafka [ERROR] Command execution failed. org.apache.commons.exec.ExecuteException: Process exited with an error: 1 (Exit value: 1) at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404) at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:166) at org.codehaus.mojo.exec.ExecMojo.executeCommandLine(ExecMojo.java:804) at org.codehaus.mojo.exec.ExecMojo.executeCommandLine(ExecMojo.java:751) at org.codehaus.mojo.exec.ExecMojo.execute(ExecMojo.java:313) at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:134) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:154) at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:146) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117) at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81) at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51) at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:309) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:194) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:107) at org.apache.maven.cli.MavenCli.execute(MavenCli.java:993) at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:345) at org.apache.maven.cli.MavenCli.main(MavenCli.java:191) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)

oswee commented 6 years ago

@iradul solution worked for me till some point. Then it broke. Not sure what caused rdkafka.pc Not found error but after spending whole evening i fixed this by creating new directory in c:\cygwin64\lib\pkgconfig and placing rdkafka.pc file there and pointing my Environment Variable to this directory. This blog lead me to solution. Mby this helps for somebody.

lagorsse commented 5 years ago

Thanks @iradul, I went throught the sames step (some step are missing but it's still quite good)

But on the last one it won't build, any idea?

D:\AMD\ProRender\rprNetworkNext\go\src\rprWorker>go build

github.com/confluentinc/confluent-kafka-go/kafka

In file included from ..\github.com\confluentinc\confluent-kafka-go\kafka\00version.go:24:0: /usr/include/librdkafka/rdkafka.h:83:47: fatal error: sys/socket.h: No such file or directory

include <sys/socket.h> / for sockaddr, .. /

                                           ^

compilation terminated.

oswee commented 5 years ago

Thanks @iradul, I went throught the sames step (some step are missing but it's still quite good)

But on the last one it won't build, any idea?

D:\AMD\ProRender\rprNetworkNext\go\src\rprWorker>go build

github.com/confluentinc/confluent-kafka-go/kafka

In file included from ..\github.com\confluentinc\confluent-kafka-go\kafka\00version.go:24:0: /usr/include/librdkafka/rdkafka.h:83:47: fatal error: sys/socket.h: No such file or directory

include <sys/socket.h> / for sockaddr, .. /

^ compilation terminated.

Did you tried to copy librdkafka.so into C:\cygwin64\lib direcory? It helped me at some point. As well remember to include @iradul mentioned files into the root of your Go project. (where the main.go lives)

CKFVL commented 5 years ago

@lagorsse

Create a folder with name sys in cygwin64/usr/include. Create a file with name as socket.h in sys folder. Copy the content from https://smealum.github.io/ctrulib/socket_8h_source.html and paste it in socket.h file.

lagorsse commented 5 years ago

Seems to work but now I got this, I tried to download nuget 0.11.5 and to replace include and lib but it keep saying this.

copy .\librdkafka.redist.0.11.5\build\native\lib\win7\x64\win7-x64-Release\v120\librdkafka.lib c:\cygwin64\lib\librdkafka.a copy .\librdkafka.redist.0.11.5\build\native\include\librdkafka\rdkafka.h c:\cygwin64\usr\include\ copy .\librdkafka.redist.0.11.5\build\native\include\librdkafka\rdkafkacpp.h c:\cygwin64\usr\include\

Is there a cache or something?

..\github.com\confluentinc\confluent-kafka-go\kafka\00version.go:44:2: error: #error "confluent-kafka-go requires librdkafka v0.11.5 or later. Install the latest version of librdkafka from the Confluent repositories, see http://docs.confluent.io/current/installation.html"

error "confluent-kafka-go requires librdkafka v0.11.5 or later. Install the latest version of librdkafka from the Confluent repositories, see http://docs.confluent.io/current/installation.html"

^~~~~

oswee commented 5 years ago

Do you have C:\cygwin64\usr\include\librdkafka directory with rdkafka.h and rdkafkacpp.h files in it?

oswee commented 5 years ago

I have this error all the time. Would like to fix it. Compiling is working fine as i do set CC=x86_64-w64-mingw32-gcc, so i can kinda live with it. But would be nice to fix. image In my Path env i have C:\cygwin64\bin where x86_64-w64-mingw32-gcc.exe lives.

Trying to update confluent-kafka-go image

oswee commented 5 years ago

So, i will put my update there. For my self in future and mby it will be helpful for others. Previously i was following @iradul instructions above. Now I removed any cygwin64 from my User Env path. Installed MinGW Selected x86_64 architecture at setup. Left other settings as default. Added C:\Program Files\mingw-w64\x86_64-8.1.0-posix-seh-rt_v6-rev0\mingw64\bin to my User Env Path. Copied librdkafka.a and librdkafkacpp.a into C:\Program Files\mingw-w64\x86_64-8.1.0-posix-seh-rt_v6-rev0\mingw64\lib Copied rdkafka.h and rdkafkacpp.h into C:\Program Files\mingw-w64\x86_64-8.1.0-posix-seh-rt_v6-rev0\mingw64\include\librdkafka Removed previously used PKG_CONFIG_PATH and its value C:\cygwin64\lib\pkgconfig from User Env. So basically eliminated any usage of cygwin64 including pkg-config. Removed rdkafka.pc from my project root. No need. I still need to have librdkafka.dll, msvcr120.dll and zlib.dll in my project root (where main.go lives). I don't no more need to use set CC=x86_64-w64-mingw32-gcc to go run main.go.

So at this point, plain Kafka command services are running and messages are committed to Kafka. But some more complex services does not run. Will make investigation and will update later does it works completely for all my projects. Any errors in VS Code are gone. Except that some services don't run (build) everything else looks fine.

oswee commented 5 years ago

So, after build and run those other projects, there is the thing. They builds. I got the .exe at the end. When i run this main.exe i got this error: image Something much better than simple exit status 3221225785 Retyping message for SEO: _The procedure entry point rd_kafka_event_CreateTopicsresult could not be located in the dynamic link library %GOPATH%\src\github.com... The rule is that project where only kafka package is used works well. But projects where kafka package + sql packages are used together doesn't run. I have assumption that this can be somehow related to 32/64 bit things as MinGW i installed previously supports booth architectures but i didn't provided kafka .dll for booth architectures.

marcofranssen commented 5 years ago

Are there any plans on what term windows is supported? I would love to contribute developing this lib and use it in my projects.

I see this issue already dates from Februari. I hope there is a plan to support the windows platform as well. It would be great for example to be able to deploy this via chocolatey.org or scoop. They both support the Nuget packaging format for deploying windows software.

oswee commented 5 years ago

Sorry for little spam, but i finally fixed this issue in its root. Now works perfectly! :) Thank you so much!!! This issue was the main reason i made this lifetime decision. Probably, other devs who work in micro-service world should think about this too. screenshot from 2018-11-08 13-53-31

vijaygurunanee commented 5 years ago

i am facing the same issues again while going through these steps and one by one explanations.... Now, for temporary reasons, i don't have any option to change OS... :( so still waiting for the same....Plz suggest if anyone has found the solution or anything is changed (or team is planning to support on windows)

neptoess commented 5 years ago

I’m on mobile, but it’s not terrible to make this work under Windows (without Cygwin). High-level: 1) Install MSYS2 2) In MSYS2, install mingw-w64-x86_64 versions of gcc and pkg-config 3) Add msys64\mingw64\bin to path 4) Copy librdkafka\rdkafka.h from librdkafka.redist NuGet package to msys64\mingw64\include\librdkafka 5) Find/replace _MSCVER with __MINGW64_\ in rdkafka.h 6) Copy librdkafka.lib from librdkafka NuGet package to msys64\mingw64\lib\librdkafka.a 7) Create rdkafka.pc under msys64\mingw64\lib\pkg-config (someone posted one earlier in this thread) 8) Make sure you have Visual C++ 2013 redistributable x64 installed 9) Find the x64 folder with librdkafka.dll and a bunch of other dlls in the librdkafka.redist NuGet package 10) Copy all the DLLs from that folder except the ones that start with msvc* to C:\Windows\System32

You should be able to go get/go build apps that use confluent-kafka-go now

humblefool007 commented 5 years ago

hi @edenhill @oswee @abhirockzz I sorted out a way to use the confluent go sdk in windows using wsl and VS code. Will document and share the steps asap.

marcofranssen commented 5 years ago

Please count me in when sharing the solution. @humblefool007

neptoess commented 5 years ago

@marcofranssen, You can try the solution I posted. It doesn’t rely on WSL.

humblefool007 commented 5 years ago

@edenhill @marcofranssen @abhirockzz @oswee .hope this helps https://medium.com/@arunagiriswaranezhilan/using-confluent-kafka-go-sdk-in-windows-wsl-vscode-ef93c443d556

edenhill commented 5 years ago

@humblefool007 Great stuff, thank you for sharing this information!

ccwillia commented 5 years ago

I am looking to build in linux (debian) but build for windows using the following:

GOOS=windows GOARCH=amd64 CGO_ENABLED=1 CC=/usr/bin/x86_64-w64-mingw32-gcc  go build -v

If i just do a go build -v there are no issues and i end up with a linux binary, but when i use the command above i get the following error 

git.target.com/sign-generate-adapter/vendor/gopkg.in/confluentinc/confluent-kafka-go.v1/kafka
# git.target.com/sign-generate-adapter/vendor/gopkg.in/confluentinc/confluent-kafka-go.v1/kafka
vendor/gopkg.in/confluentinc/confluent-kafka-go.v1/kafka/00version.go:24:10: fatal error: librdkafka/rdkafka.h: No such file or directory
 #include <librdkafka/rdkafka.h>
          ^~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
marcofranssen commented 5 years ago

@ccwillia that is exactly the same issue I'm having on windows regarding the librdkafka

ccwillia commented 5 years ago

@marcofranssen are you trying to build on windows or linux? I need to be able to build on linux but the output needs to be a windows binary. If you are trying to build on windows i was able to get that working with cygwin64

neptoess commented 4 years ago

I decided to play around with confluent-kafka-go under Windows (not WSL) some more. While my previous instructions sufficed, I've decided to streamline things a bit to make it easier if I or anyone else chooses to do this in the future. These steps should apply to pretty much any library that has to built from source with cgo on Windows. This should all work for v1.3.0 (any files attached came from librdkafka.redist v1.3.0)

  1. Install the x86_64 version of MSYS2 (https://www.msys2.org/) (I used all defaults during the install)
  2. After the installation, the MSYS2 MSYS shell should already be running. If not, launch it
  3. Run pacman -Syu
  4. Close the MSYS window when prompted to (just click OK on the warning) image
  5. Open MSYS2 MSYS again
  6. Run pacman -Syu again
  7. Run pacman -S mingw-w64-x86_64-gcc mingw-w64-x86_64-pkg-config
  8. Add C:\msys64\mingw64\bin to your path image image image
  9. Extract librdkafka-mingw64-v1.3.0.zip to C:\msys64\mingw64

You should now be able to go build and confluent-kafka-go will compile successfully, but you should probably read on.

Runtime DLLs must be distributed somehow, or your app won't run.

  1. The easiest way to do this is to simply extract librdkafka-win-x64-v.1.3.0.zip into the same directory as your .exe file. This must be done for every EXE you build that depends on confluent-kafka-go.
  2. Another way to do this is to install the Visual C++ 2013 redistributable (https://aka.ms/highdpimfc2013x64enu) and copy the files from the above zip file (except the ones that start with ms*) to C:\Windows\System32. If you develop a lot of confluent-kafka-go apps on Windows, and you never plan to deploy on Windows, this is probably the path of least headaches.
rwsegura commented 4 years ago

@neptoess I am attempting your method but it seems on compilation I am getting undefined reference to errors

C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/9.3.0/../../../../x86_64-w64-mingw32/bin/ld.exe: $WORK\b122\_x003.o: in function `_cgo_6f41379583e6_Cfunc_rd_kafka_AlterConfigs':
/tmp/go-build/cgo-gcc-prolog:117: undefined reference to `rd_kafka_AlterConfigs'
neptoess commented 4 years ago

@rwsegura , The files I included were all based on librdkafka 1.3.0, where the latest confluent-kafka-go is targeting 1.4.0. If you look at both of my comments on this thread, you should be able to fairly easily get 1.4.0 linkable under MinGW.

If you can wait a bit, v1.5.0 will make this whole process much easier, since we got a PR merged that enables a MinGW build, which opens the door for shipping a static library with confluent-kafka-go.

Let me know if you hit any snags grabbing the NuGet package, modifying rdkafka.h, and copying the files around. I'll be glad to help.

rwsegura commented 4 years ago

@neptoess - Thank you this resolved my issue :)

ligoo commented 4 years ago

I followed every step provided by @neptoess in his well explained posts. I've adapted the method to make it for v1.4.2 (modification of rdkafka.pc, copying files around). Seems I still can't make it work properly:

In file included from github.com\confluentinc\confluent-kafka-go\kafka\00version.go:24:
./librdkafka/rdkafka.h:83:10: fatal error: sys/socket.h: No such file or directory
   83 | #include <sys/socket.h> /* for sockaddr, .. */
      |          ^~~~~~~~~~~~~~
compilation terminated.

@rwsegura did you make it work for 1.4.0 ? Any extra steps you had to do ?

neptoess commented 4 years ago

@ligoo , It looks like you forgot to modify the #ifdef _MSC_VER checks in the rdkafka.h you copied to your mingw include directory to be either #ifdef __MINGW64__ or #ifdef _WIN32

ligoo commented 4 years ago

@neptoess I replaced 2 occurences of #ifdef _MSC_VER by #ifdef __MINGW64__ on line 58 and 2025. That's why I don't get it. EDIT: I'm not sure why it doesn't read the right file rdkafka.h in the C:\msys64\mingw64\include\librdkafka directory but takes the one in C:\dev\go\src\github.com\confluentinc\confluent-kafka-go\kafka\librdkafka. So I modified the file directly inside the confluentinc directory to see what happens and it does start compiling but he stops with thes kinds of errors:

# github.com/confluentinc/confluent-kafka-go/kafka
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: $WORK\b116\_x003.o: in function `_cgo_58d8fda2c605_Cfunc_rd_kafka_AdminOptions_destroy':
/tmp/go-build/cgo-gcc-prolog:101: undefined reference to `__imp_rd_kafka_AdminOptions_destroy'

Any clue what I'm missing here ?

ligoo commented 4 years ago

So to recap the steps: 1) follow @neptoess steps 1-8 2) copy from the NuGet package downloaded from https://www.nuget.org/packages/librdkafka.redist/ (version 1.4.2) \build\native\include\librdkafka\rdkafka.h => C:\msys64\mingw64\include\librdkafka\rdkafka.h 3) modify C:\msys64\mingw64\include\librdkafka\rdkafka.h and replace #ifdef _MSC_VER by #ifdef __MINGW64__ (2 times) 4) copy from the NuGet package \build\native\lib\win\x64\win-x64-Release\v120\librdkafka.lib => C:\msys64\mingw64\lib\librdkafka.a 5) put in C:\msys64\mingw64\lib\pkgconfig\lib\pkconfig\rdkafka.pc:

prefix=c:/
libdir=c:/msys64/mingw64/lib/
includedir=c:/msys64/mingw64/include

Name: librdkafka
Description: The Apache Kafka C/C++ library
Version: 1.4.2
Cflags: -I${includedir}
Libs: -L${libdir} -lrdkafka
Libs.private: -lssl -lcrypto -lcrypto -lz -ldl -lpthread -lrt

6) install Visual C++ 2013 redistributable (https://aka.ms/highdpimfc2013x64enu) 7) copy from the NuGet package \runtimes\win-x64\native\*.dll into C:\Windows\System32 except msv*.dll (librdkafka.dll, librdkafkacpp.dll, libzstd.dll, zlib.dll)

When trying to go build the answer was: ./librdkafka/rdkafka.h:83:10: fatal error: sys/socket.h: No such file or directory => Modification of rdkafka.h inside the confluentinc and replaced #ifdef _MSC_VER by #ifdef __MINGW64__

Result as described on the previous post:

# gopkg.in/confluentinc/confluent-kafka-go.v1/kafka
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: $WORK\b173\_x003.o: in function `_cgo_686f979efb04_Cfunc_rd_kafka_AdminOptions_destroy':
/tmp/go-build/cgo-gcc-prolog:101: undefined reference to `__imp_rd_kafka_AdminOptions_destroy'
neptoess commented 4 years ago

@ligoo , I’m not sure if @rwsegura ran into this, but my only guess would be doing a go get -u to make sure you’re using the newest confluent-kafka-go. The error means that the go/cgo code is looking for a function that isn’t defined in your librdkafka.a

ligoo commented 4 years ago

@neptoess I tried the go get -u without any luck. I've seens that the error messages are all comming from undefined reference to __imp_rd_kafka_xxxyyyzzz' which are called with something like this

__IMPORT_DESCRIPTOR_librdkafka __NULL_IMPORT_DESCRIPTOR librdkafka_NULL_THUNK_DATA __imp_rd_kafka_version rd_kafka_version

from librdkafka.a. I'm not sure about why there is an _imp_ prefix in this file. @rwsegura error message was refering to rd_kafka_AlterConfigs without this prefix. So it seems to be kinda different from what I got here. I've looked up in the .so file from the linux \runtimes\linux-x64\native\librdkafka.so and there is no mention of that _imp_ in there. In your previous librdkafka-mingw64-v1.3.0.zip pack file the librdkafka.a did not contain the _imp_ prefix. It's comming from >=1.4.0 version and only in the windows version of it. I'm kinda curious how @rwsegura made it work with 1.4.0 because the files share these prefix similarities. @edenhill Any insight on this ?

ligoo commented 4 years ago

I tried everything I could to make it work with 1.4.0 and 1.4.2 without any success. For now I'll just fall back to 1.3.0 and it works just fine with your method @neptoess 🎉 If someone has some success with >=1.40 please share with the commnunity.

neptoess commented 4 years ago

@ligoo It shouldn’t be too much longer for librdkafka v1.5.0. Once that’s released I can put the effort in to get a MinGW static lib in confluent-kafka-go, and this will all get much simpler.

rwsegura commented 4 years ago

@ligoo - I was using 1.4.0 but if I remember correctly there was a couple of .h files in the librdkafka I had modified. When I get back to my PC later today I'll if I see anything related to a prefix and what else I had done.

neptoess commented 4 years ago

@ligoo , Gave it a shot. Looks like, for v1.4.2, I had to, like you, modify the rdkafka.h in go\src\github.com\confluentinc\confluent-kafka-go\kafka\librdkafka. I also modified 00version.go by adding #cgo LDFLAGS: -lrdkafka before #include <librdkafka/rdkafka.h>.

This tells the MinGW linker to link its librdkafka.a, since none of the static libraries provided with confluent-kafka-go are going to be linked