search

matrix-org / olm

An implementation of the Double Ratchet cryptographic ratchet in C++/C
https://gitlab.matrix.org/matrix-org/olm
63 stars 9 forks source link

Cannot install python-olm on Windows via pip (Py38 amd64) #25

Open danielniccoli opened 4 years ago

danielniccoli commented 4 years ago 
C:\>pip install python-olm
Collecting python-olm
  Using cached https://files.pythonhosted.org/packages/d4/a4/1face47e65118d7c52726dfa305410a96bc4a0c6f3f99c90bc7104aebf21/python-olm-3.1.3.tar.gz
Requirement already satisfied: cffi>=1.0.0 in c:\users\me\appdata\local\programs\python\python38\lib\site-packages (from python-olm) (1.13.2)
Requirement already satisfied: future in c:\users\me\appdata\local\programs\python\python38\lib\site-packages (from python-olm) (0.18.2)
Requirement already satisfied: pycparser in c:\users\me\appdata\local\programs\python\python38\lib\site-packages (from cffi>=1.0.0->python-olm) (2.19)
Building wheels for collected packages: python-olm
  Building wheel for python-olm (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: 'c:\users\me\appdata\local\programs\python\python38\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\me\\AppData\\Local\\Temp\\pip-install-1myzii8l\\python-olm\\setup.py'"'"'; __file__='"'"'C:\\Users\\me\\AppData\\Local\\Temp\\pip-install-1myzii8l\\python-olm\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d 'C:\Users\me\AppData\Local\Temp\pip-wheel-vv1l60m8' --python-tag cp38
       cwd: C:\Users\me\AppData\Local\Temp\pip-install-1myzii8l\python-olm\
  Complete output (30 lines):
  'make' is not recognized as an internal or external command,
  operable program or batch file.
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build\lib.win-amd64-3.8
  creating build\lib.win-amd64-3.8\olm
  copying olm\account.py -> build\lib.win-amd64-3.8\olm
  copying olm\group_session.py -> build\lib.win-amd64-3.8\olm
  copying olm\pk.py -> build\lib.win-amd64-3.8\olm
  copying olm\sas.py -> build\lib.win-amd64-3.8\olm
  copying olm\session.py -> build\lib.win-amd64-3.8\olm
  copying olm\utility.py -> build\lib.win-amd64-3.8\olm
  copying olm\_compat.py -> build\lib.win-amd64-3.8\olm
  copying olm\_finalize.py -> build\lib.win-amd64-3.8\olm
  copying olm\__init__.py -> build\lib.win-amd64-3.8\olm
  copying olm\__version__.py -> build\lib.win-amd64-3.8\olm
  running build_ext
  generating cffi module 'build\\temp.win-amd64-3.8\\Release\\_libolm.c'
  creating build\temp.win-amd64-3.8
  creating build\temp.win-amd64-3.8\Release
  building '_libolm' extension
  creating build\temp.win-amd64-3.8\Release\build
  creating build\temp.win-amd64-3.8\Release\build\temp.win-amd64-3.8
  creating build\temp.win-amd64-3.8\Release\build\temp.win-amd64-3.8\Release
  C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.24.28314\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Ic:\users\me\appdata\local\programs\python\python38\include -Ic:\users\me\appdata\local\programs\python\python38\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.24.28314\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt" /Tcbuild\temp.win-amd64-3.8\Release\_libolm.c /Fobuild\temp.win-amd64-3.8\Release\build\temp.win-amd64-3.8\Release\_libolm.obj -I../include
  _libolm.c
  build\temp.win-amd64-3.8\Release\_libolm.c(493): fatal error C1083: Cannot open include file: 'olm/olm.h': No such file or directory
  error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.24.28314\\bin\\HostX86\\x64\\cl.exe' failed with exit status 2
  ----------------------------------------
  ERROR: Failed building wheel for python-olm
  Running setup.py clean for python-olm
Failed to build python-olm
Installing collected packages: python-olm
    Running setup.py install for python-olm ... error
    ERROR: Command errored out with exit status 1:
     command: 'c:\users\me\appdata\local\programs\python\python38\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\me\\AppData\\Local\\Temp\\pip-install-1myzii8l\\python-olm\\setup.py'"'"'; __file__='"'"'C:\\Users\\me\\AppData\\Local\\Temp\\pip-install-1myzii8l\\python-olm\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\me\AppData\Local\Temp\pip-record-kcobwouw\install-record.txt' --single-version-externally-managed --compile
         cwd: C:\Users\me\AppData\Local\Temp\pip-install-1myzii8l\python-olm\
    Complete output (30 lines):
    'make' is not recognized as an internal or external command,
    operable program or batch file.
    running install
    running build
    running build_py
    creating build
    creating build\lib.win-amd64-3.8
    creating build\lib.win-amd64-3.8\olm
    copying olm\account.py -> build\lib.win-amd64-3.8\olm
    copying olm\group_session.py -> build\lib.win-amd64-3.8\olm
    copying olm\pk.py -> build\lib.win-amd64-3.8\olm
    copying olm\sas.py -> build\lib.win-amd64-3.8\olm
    copying olm\session.py -> build\lib.win-amd64-3.8\olm
    copying olm\utility.py -> build\lib.win-amd64-3.8\olm
    copying olm\_compat.py -> build\lib.win-amd64-3.8\olm
    copying olm\_finalize.py -> build\lib.win-amd64-3.8\olm
    copying olm\__init__.py -> build\lib.win-amd64-3.8\olm
    copying olm\__version__.py -> build\lib.win-amd64-3.8\olm
    running build_ext
    generating cffi module 'build\\temp.win-amd64-3.8\\Release\\_libolm.c'
    creating build\temp.win-amd64-3.8
    creating build\temp.win-amd64-3.8\Release
    building '_libolm' extension
    creating build\temp.win-amd64-3.8\Release\build
    creating build\temp.win-amd64-3.8\Release\build\temp.win-amd64-3.8
    creating build\temp.win-amd64-3.8\Release\build\temp.win-amd64-3.8\Release
    C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.24.28314\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -Ic:\users\me\appdata\local\programs\python\python38\include -Ic:\users\me\appdata\local\programs\python\python38\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.24.28314\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt" /Tcbuild\temp.win-amd64-3.8\Release\_libolm.c /Fobuild\temp.win-amd64-3.8\Release\build\temp.win-amd64-3.8\Release\_libolm.obj -I../include
    _libolm.c
    build\temp.win-amd64-3.8\Release\_libolm.c(493): fatal error C1083: Cannot open include file: 'olm/olm.h': No such file or directory
    error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\Tools\\MSVC\\14.24.28314\\bin\\HostX86\\x64\\cl.exe' failed with exit status 2
    ----------------------------------------
ERROR: Command errored out with exit status 1: 'c:\users\me\appdata\local\programs\python\python38\python.exe' -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\me\\AppData\\Local\\Temp\\pip-install-1myzii8l\\python-olm\\setup.py'"'"'; __file__='"'"'C:\\Users\\me\\AppData\\Local\\Temp\\pip-install-1myzii8l\\python-olm\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record 'C:\Users\me\AppData\Local\Temp\pip-record-kcobwouw\install-record.txt' --single-version-externally-managed --compile Check the logs for full command output.
poljar commented 4 years ago

The Python bindings require the C lib and C headers to be installed beforehand. @turt2live could you lend a hand in Windows land?

I believe the error about the make file is safely ignored even though it shouldn't be there, this should be fixed by next release.

Thanks for the report.

danielniccoli commented 4 years ago

The Python bindings require the C lib and C headers to be installed beforehand

Compiling on Windows is a terrible user experience for people who develope in Python and have no experience with C/C++. The knowledge isn't there, the build tools aren't there, and where are we supposed to get C libs/headers or place them? For example, I had to install the VS C++ build tools, which already required me to install a 4 GB package of Visual Studio on my computer only to compile a small module (which then still didn't work, as shown above).

The best experience for any Windows user is, when they find a binary wheel uploaded to Pypi.

This system of conflating build-install is slow, hard to maintain, and hinders innovation in both build systems and installers.

Wheel attempts to remedy these problems by providing a simpler interface between the build system and the installer. The wheel binary package format frees installers from having to know about the build system, saves time by amortizing compile time over many installations, and removes the need to install a build system in the target environment. https://www.python.org/dev/peps/pep-0427/

Would be great if that could happen when you release the next version. 💚

richvdh commented 4 years ago

Compiling on Windows is a terrible user experience for people who develope in Python ...

yes it is, particularly when you don't have a windows machine to build on... which is a big reason we don't ship windows binaries.

poljar commented 4 years ago

We might want to let CI build statically linked wheels using something like cibuildwheel.

CI would in this case push releases to PyPi as well, but that requires CI to be set up and I forgot what our stance on the legal side of things is with regards to this.

merlink01 commented 4 years ago

Anybody got it compiled?

dirkbike commented 4 years ago

It builds under mingw-w64 with a bit of effort, if someone cleans up the build files to handle the environment the process would be easy for most Windows users. Here's a how to get it working in the meantime:

Setup the mingw-w64 environment with updated dependencies and make sure you are in the 64-bit MINGW shell (not MSYS). Also make sure to use mingw's pip.

pacman -Syu mingw-w64-x86_64-python-pip libcrypt-devel libffi-devel
/mingw64/bin/pip install --upgrade pip
/mingw64/bin/pip install --upgrade setuptools
/mingw64/bin/pip install --upgrade wheel
/mingw64/bin/pip install --upgrade cffi

Get the latest olm source and build the static library instead of the default dynamic library. You also need to explicitly tell cmake to use the standard c++ library (libstdc++.a) since it doesn't assume this on it's own.

git clone https://gitlab.matrix.org/matrix-org/olm.git
cd olm
cmake . -Bbuild -DBUILD_SHARED_LIBS=NO -DCMAKE_CXX_STANDARD_LIBRARIES="-lstdc++"
cmake --build build

Once the build is done you should have the olm static library (libolm.a) in your build directory. Test it by running ctest while inside the build/tests directory and all tests should pass.

pushd build/tests
ctest .
popd

The next step is to build the Python wheel from the python directory.

cd python
python setup.py bdist_wheel

This will fail because the generated compile command for _libolm-cpython-38.dll misses the libstdc++.a requirement and fails to put the library paths (-L arguments) before the libraries themselves (-l arguments). You also need to add a -static argument to build the shared library without other shared library dependencies. It's good practice (may not be necessary) to have it reference the Python shared library where you will use the wheel instead of mingw's Python. On my system that was C:\Python38\python38.dll and you can specify that by replacing -LC:/msys64/mingw64/lib/python3.8/config-3.8 and -lpython3.8 with the arguments -L/c/python38 and -l:python38.dll (note the colon is needed to specify the exact file name). The fixed compile command looked like this on my system.

/c/msys64/mingw64/bin/x86_64-w64-mingw32-gcc.exe -shared -static -Wl,--enable-auto-image-base -pipe -s build/temp.mingw-3.8/build/temp.mingw-3.8/_libolm.o build/temp.mingw-3.8/build/temp.mingw-3.8/_libolm-cpython-38.def -LC:/msys64/mingw64/lib -L../build -L/c/python38 -lolm -l:python38.dll -lm -lversion -lshlwapi -lstdc++ -o build/lib.mingw-3.8/_libolm-cpython-38.dll

After re-running the compile command you can continue building the Python wheel.

python setup.py bdist_wheel

You should now have a completed wheel (python_olm-3.1.4-cp38-cp38-mingw.whl) in your dist directory, but it specifies mingw as the platform instead of win_amd64 and it's expecting the shared library to be _libolm.dll instead of _libolm-cpython-38.dll. Both of these issues can be fixed by re-packaging the wheel.

cd dist
wheel unpack python_olm-3.1.4-cp38-cp38-mingw.whl
mv python_olm-3.1.4/_libolm-cpython-38.dll python_olm-3.1.4/_libolm.pyd
perl -i -pe 's/cp38-cp38-mingw/cp38-cp38-win_amd64/' python_olm-3.1.4/python_olm-3.1.4.dist-info/WHEEL
wheel pack python_olm-3.1.4

If all of that worked you'll end up with a wheel called python_olm-3.1.4-cp38-cp38-win_amd64.whl that you can install like any other wheel.

padmalcom commented 3 years ago

@dirkbike Would it be possible to upload the wheel for cp38 and win64? This issue still seems to be unsolved.

ghost commented 2 years ago

This issue is still not resolved. Apart from the method @dirkbike mentioned, is there an easier way to install python-olm on windows? It would be great if we'd get the .whl file from the aforementioned method. Thanks

markusf1895 commented 2 years ago

Why not use wsl2? It works fine for me.

mrx23dot commented 1 month ago

Please add prebuilt libs or Windows onto pypi.