search

confluentinc / confluent-kafka-python

Confluent's Kafka Python Client
http://docs.confluent.io/current/clients/confluent-kafka-python
Other
111 stars 895 forks source link

Cannot install confluent-kafka #805

Closed dvojnic closed 3 years ago

dvojnic commented 4 years ago

When I run the install command pip install confluent-kafka like specified in the README file, I get the following error:

fatal error C1083: Cannot open include file: 'librdkafka/rdkafka.h': No such file or directory error: command 'C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.24.28314\bin\HostX86\x86\cl.exe' failed with exit status 2

My python version is 3.8.2 and my pip version is 19.2.3 Not sure if this is needed information, but I installed python in a specific folder, not the default one.

Before this error, I had an error which said that Microsoft Visual C++ 14.0 is required, so I got it with the build tools. After installing Microsoft Visual Studio Build Tools 2019, I received the above mentioned bold error. I searched online for solutions and saw that someone suggested getting Build Tools 2013 version, however that was suggested 2 years ago and didn't work when I tried it out.

Help will be much appreciated!

rnpridgeon commented 4 years ago

This is an outstanding issue we plan to address after the upcoming release. We use a deprecated portion of the C Python API which causes issues with our wheel builds on appveyor.

binnyc commented 4 years ago

Am running into same issue. Any update when the fix for this is expected. Thank you!

edenhill commented 4 years ago

Confluent-kafka-python with Python 3.8 is not yet supported on Windows, we'll look into this during Q3.

Paul-bot commented 4 years ago

Confluent-kafka-python with Python 3.8 is not yet supported on Windows, we'll look into this during Q3.

Hi. I'm interested to know if we're any closer to a fix for this, as we're looking to upgrade to 3.8? Much appreciated!

alieismy commented 4 years ago

Confluent-kafka-python with Python 3.8 is not yet supported on Windows, we'll look into this during Q3.

Hi. I'm interested to know if we're any closer to a fix for this, as we're looking to upgrade to 3.8? Much appreciated!

Same problem as above.

steverawlins-zebra commented 3 years ago

What is the work-around?

edenhill commented 3 years ago

v1.6.0 will have windows wheels for 3.8

SaiSudheerK commented 3 years ago

Hi, I am facing the same issue. When I tried to install confluent-Kafka in my python 3.10, the following error is showing fatal error C1083: Cannot open include file: 'librdkafka/rdkafka.h': No such file or directory. Before this error, I had an error that said that Microsoft Visual C++ 14.0 is required, so I got it with the build tools. After installing Microsoft Visual Studio Build Tools 2019, I received the above-mentioned bold error.

schnells commented 2 years ago

Confluent-kafka-python with Python 3.8 is not yet supported on Windows, we'll look into this during Q3.

Still the same issue in 2022. Any updates / workarounds? Current workaround is to run the python applications inside docker. But I'm currently working on a project which requires UI stuff, so containers wont help

edenhill commented 2 years ago

We provide Py 3.7, 3.8 and 3.9 wheels for Windows.

nimdeveloper commented 2 years ago

I'm using Python 3.10.1, and installed librdkafka using vcpkg, but when I run command pip install confluent-kafka, I get this error: C:\Users\Nimdeveloper\AppData\Local\Temp\pip-install-vbahpbo_\confluent-kafka_c3499a36ffaa4abbaa6871860b86e153\src\confluent_kafka\src\confluent_kafka.h(23): fatal error C1083: Cannot open include file: 'librdkafka/rdkafka.h': No such file or directory After searching, I found this command: pip install --global-option=build_ext --global-option="-LC:\dev\vcpkg\packages\librdkafka_x86-windows\lib" --global-option="-IC:\dev\vcpkg\packages\librdkafka_x86-windows\include" confluent-kafka and, get new error: C:\Users\Nimdeveloper\AppData\Local\Temp\pip-install-qkduyfnz\confluent-kafka_de793e8ea9f148cc9f6de18ad779073d\src\confluent_kafka\src\Admin.c : fatal error C1083: Cannot open compiler generated file: '': Invalid argument version of confluent-kafka is 1.8.2. any solutions for this error?

schnells commented 2 years ago

I'm using Python 3.10.1, and installed librdkafka using vcpkg, but when I run command pip install confluent-kafka, I get this error: C:\Users\Nimdeveloper\AppData\Local\Temp\pip-install-vbahpbo_\confluent-kafka_c3499a36ffaa4abbaa6871860b86e153\src\confluent_kafka\src\confluent_kafka.h(23): fatal error C1083: Cannot open include file: 'librdkafka/rdkafka.h': No such file or directory After searching, I found this command: pip install --global-option=build_ext --global-option="-LC:\dev\vcpkg\packages\librdkafka_x86-windows\lib" --global-option="-IC:\dev\vcpkg\packages\librdkafka_x86-windows\include" confluent-kafka and, get new error: C:\Users\Nimdeveloper\AppData\Local\Temp\pip-install-qkduyfnz\confluent-kafka_de793e8ea9f148cc9f6de18ad779073d\src\confluent_kafka\src\Admin.c : fatal error C1083: Cannot open compiler generated file: '': Invalid argument version of confluent-kafka is 1.8.2. any solutions for this error?

Downgrade Python to 3.9 works

nimdeveloper commented 2 years ago

Downgrade Python to 3.9 works

Thanks, it worked!

SamuelJansen commented 2 years ago

Such a huge information (like: only works on python 3.9) should be stick into setup.py itself...

SamuelJansen commented 2 years ago

pip install --global-option=build_ext --global-option="-LC:\dev\vcpkg\packages\librdkafka_x86-windows\lib" --global-option="-IC:\dev\vcpkg\packages\librdkafka_x86-windows\include" confluent-kafka

Doesn't seem to work on python3.9 64 bits

C:\[[user-path]]>pip3.9 install --global-option=build_ext --global-option="-LC:\dev\vcpkg\packages\librdkafka_x86-windows\lib" --global-option="-IC:\dev\vcpkg\packages\librdkafka_x86-windows\include" confluent-kafka
WARNING: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
Collecting confluent-kafka
  Using cached confluent-kafka-1.8.2.tar.gz (104 kB)
  Preparing metadata (setup.py) ... done
Skipping wheel build for confluent-kafka, due to binaries being disabled for it.
Installing collected packages: confluent-kafka
  Running setup.py install for confluent-kafka ... error
  error: subprocess-exited-with-error

  × Running setup.py install for confluent-kafka did not run successfully.
  │ exit code: 1
  ╰─> [19 lines of output]
      running build_ext
      building 'confluent_kafka.cimpl' extension
      creating build
      creating build\temp.win-amd64-3.9
      creating build\temp.win-amd64-3.9\Release
      creating build\temp.win-amd64-3.9\Release\Users
      creating build\temp.win-amd64-3.9\Release\[[user-path]]
      creating build\temp.win-amd64-3.9\Release\[[user-path]]\AppData
      creating build\temp.win-amd64-3.9\Release\[[user-path]]\AppData\Local
      creating build\temp.win-amd64-3.9\Release\[[user-path]]\AppData\Local\Temp
      creating build\temp.win-amd64-3.9\Release\[[user-path]]\AppData\Local\Temp\pip-install-4pmbyfb2
      creating build\temp.win-amd64-3.9\Release\[[user-path]]\AppData\Local\Temp\pip-install-4pmbyfb2\confluent-kafka_0905df29abb740f3907be455af4367e0
      creating build\temp.win-amd64-3.9\Release\[[user-path]]\AppData\Local\Temp\pip-install-4pmbyfb2\confluent-kafka_0905df29abb740f3907be455af4367e0\src
      creating build\temp.win-amd64-3.9\Release\[[user-path]]\AppData\Local\Temp\pip-install-4pmbyfb2\confluent-kafka_0905df29abb740f3907be455af4367e0\src\confluent_kafka
      creating build\temp.win-amd64-3.9\Release\[[user-path]]\AppData\Local\Temp\pip-install-4pmbyfb2\confluent-kafka_0905df29abb740f3907be455af4367e0\src\confluent_kafka\src
      C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC:\dev\vcpkg\packages\librdkafka_x86-windows\include -IC:\[[user-path]]\AppData\Local\Programs\Python\Python39\include -IC:\[[user-path]]\AppData\Local\Programs\Python\Python39\include -IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\include -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt /TcC:\[[user-path]]\AppData\Local\Temp\pip-install-4pmbyfb2\confluent-kafka_0905df29abb740f3907be455af4367e0\src\confluent_kafka\src\Admin.c /Fobuild\temp.win-amd64-3.9\Release\[[user-path]]\AppData\Local\Temp\pip-install-4pmbyfb2\confluent-kafka_0905df29abb740f3907be455af4367e0\src\confluent_kafka\src\Admin.obj
      Admin.c
      C:\[[user-path]]\AppData\Local\Temp\pip-install-4pmbyfb2\confluent-kafka_0905df29abb740f3907be455af4367e0\src\confluent_kafka\src\confluent_kafka.h(23): fatal error C1083: NÆo ‚ poss¡vel abrir arquivo incluir: 'librdkafka/rdkafka.h': No such file or directory
      error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> confluent-kafka

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
SamuelJansen commented 2 years ago

fixing confluent version (to the above cited) doesn't work either

C:\[[user-path]]>pip3.9 install --global-option=build_ext --global-option="-LC:\dev\vcpkg\packages\librdkafka_x86-windows\lib" --global-option="-IC:\dev\vcpkg\packages\librdkafka_x86-windows\include" confluent-kafka==1.8.2
WARNING: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
Collecting confluent-kafka==1.8.2
  Using cached confluent-kafka-1.8.2.tar.gz (104 kB)
  Preparing metadata (setup.py) ... done
Skipping wheel build for confluent-kafka, due to binaries being disabled for it.
Installing collected packages: confluent-kafka
  Running setup.py install for confluent-kafka ... error
  error: subprocess-exited-with-error

  × Running setup.py install for confluent-kafka did not run successfully.
  │ exit code: 1
  ╰─> [19 lines of output]
      running build_ext
      building 'confluent_kafka.cimpl' extension
      creating build
      creating build\temp.win-amd64-3.9
      creating build\temp.win-amd64-3.9\Release
      creating build\temp.win-amd64-3.9\Release\Users
      creating build\temp.win-amd64-3.9\Release\[[user-path]]
      creating build\temp.win-amd64-3.9\Release\[[user-path]]\AppData
      creating build\temp.win-amd64-3.9\Release\[[user-path]]\AppData\Local
      creating build\temp.win-amd64-3.9\Release\[[user-path]]\AppData\Local\Temp
      creating build\temp.win-amd64-3.9\Release\[[user-path]]\AppData\Local\Temp\pip-install-in3uzktu
      creating build\temp.win-amd64-3.9\Release\[[user-path]]\AppData\Local\Temp\pip-install-in3uzktu\confluent-kafka_e6955e21fdf1420084c5d87611dec5d3
      creating build\temp.win-amd64-3.9\Release\[[user-path]]\AppData\Local\Temp\pip-install-in3uzktu\confluent-kafka_e6955e21fdf1420084c5d87611dec5d3\src
      creating build\temp.win-amd64-3.9\Release\[[user-path]]\AppData\Local\Temp\pip-install-in3uzktu\confluent-kafka_e6955e21fdf1420084c5d87611dec5d3\src\confluent_kafka
      creating build\temp.win-amd64-3.9\Release\[[user-path]]\AppData\Local\Temp\pip-install-in3uzktu\confluent-kafka_e6955e21fdf1420084c5d87611dec5d3\src\confluent_kafka\src
      C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC:\dev\vcpkg\packages\librdkafka_x86-windows\include -IC:\[[user-path]]\AppData\Local\Programs\Python\Python39\include -IC:\[[user-path]]\AppData\Local\Programs\Python\Python39\include -IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\include -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt /TcC:\[[user-path]]\AppData\Local\Temp\pip-install-in3uzktu\confluent-kafka_e6955e21fdf1420084c5d87611dec5d3\src\confluent_kafka\src\Admin.c /Fobuild\temp.win-amd64-3.9\Release\[[user-path]]\AppData\Local\Temp\pip-install-in3uzktu\confluent-kafka_e6955e21fdf1420084c5d87611dec5d3\src\confluent_kafka\src\Admin.obj
      Admin.c
      C:\[[user-path]]\AppData\Local\Temp\pip-install-in3uzktu\confluent-kafka_e6955e21fdf1420084c5d87611dec5d3\src\confluent_kafka\src\confluent_kafka.h(23): fatal error C1083: NÆo ‚ poss¡vel abrir arquivo incluir: 'librdkafka/rdkafka.h': No such file or directory
      error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> confluent-kafka

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
nimdeveloper commented 2 years ago

fixing confluent version (to the above cited) doesn't work either

C:\[[user-path]]>pip3.9 install --global-option=build_ext --global-option="-LC:\dev\vcpkg\packages\librdkafka_x86-windows\lib" --global-option="-IC:\dev\vcpkg\packages\librdkafka_x86-windows\include" confluent-kafka==1.8.2
WARNING: Disabling all use of wheels due to the use of --build-option / --global-option / --install-option.
Collecting confluent-kafka==1.8.2
  Using cached confluent-kafka-1.8.2.tar.gz (104 kB)
  Preparing metadata (setup.py) ... done
Skipping wheel build for confluent-kafka, due to binaries being disabled for it.
Installing collected packages: confluent-kafka
  Running setup.py install for confluent-kafka ... error
  error: subprocess-exited-with-error

  × Running setup.py install for confluent-kafka did not run successfully.
  │ exit code: 1
  ╰─> [19 lines of output]
      running build_ext
      building 'confluent_kafka.cimpl' extension
      creating build
      creating build\temp.win-amd64-3.9
      creating build\temp.win-amd64-3.9\Release
      creating build\temp.win-amd64-3.9\Release\Users
      creating build\temp.win-amd64-3.9\Release\[[user-path]]
      creating build\temp.win-amd64-3.9\Release\[[user-path]]\AppData
      creating build\temp.win-amd64-3.9\Release\[[user-path]]\AppData\Local
      creating build\temp.win-amd64-3.9\Release\[[user-path]]\AppData\Local\Temp
      creating build\temp.win-amd64-3.9\Release\[[user-path]]\AppData\Local\Temp\pip-install-in3uzktu
      creating build\temp.win-amd64-3.9\Release\[[user-path]]\AppData\Local\Temp\pip-install-in3uzktu\confluent-kafka_e6955e21fdf1420084c5d87611dec5d3
      creating build\temp.win-amd64-3.9\Release\[[user-path]]\AppData\Local\Temp\pip-install-in3uzktu\confluent-kafka_e6955e21fdf1420084c5d87611dec5d3\src
      creating build\temp.win-amd64-3.9\Release\[[user-path]]\AppData\Local\Temp\pip-install-in3uzktu\confluent-kafka_e6955e21fdf1420084c5d87611dec5d3\src\confluent_kafka
      creating build\temp.win-amd64-3.9\Release\[[user-path]]\AppData\Local\Temp\pip-install-in3uzktu\confluent-kafka_e6955e21fdf1420084c5d87611dec5d3\src\confluent_kafka\src
      C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC:\dev\vcpkg\packages\librdkafka_x86-windows\include -IC:\[[user-path]]\AppData\Local\Programs\Python\Python39\include -IC:\[[user-path]]\AppData\Local\Programs\Python\Python39\include -IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.29.30133\include -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.19041.0\cppwinrt /TcC:\[[user-path]]\AppData\Local\Temp\pip-install-in3uzktu\confluent-kafka_e6955e21fdf1420084c5d87611dec5d3\src\confluent_kafka\src\Admin.c /Fobuild\temp.win-amd64-3.9\Release\[[user-path]]\AppData\Local\Temp\pip-install-in3uzktu\confluent-kafka_e6955e21fdf1420084c5d87611dec5d3\src\confluent_kafka\src\Admin.obj
      Admin.c
      C:\[[user-path]]\AppData\Local\Temp\pip-install-in3uzktu\confluent-kafka_e6955e21fdf1420084c5d87611dec5d3\src\confluent_kafka\src\confluent_kafka.h(23): fatal error C1083: NÆo ‚ poss¡vel abrir arquivo incluir: 'librdkafka/rdkafka.h': No such file or directory
      error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.29.30133\\bin\\HostX86\\x64\\cl.exe' failed with exit code 2
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> confluent-kafka

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

with python 3.9 I just executed the install command pip install confluent-kafka and it worked, but still installation process is not clear. I think it's better to use it on docker with Linux, not sure.