Open nkaenzig opened 3 years ago
Hi @nkaenzig , thanks for the report!
Can you try to run python setup.py install
(use the python interpreter you want to install the library for, so maybe python3
for example) from the python
folder? You should expect the issue to happen again. Can you tell me if this builds correctly: pip install -r dev-requirements.py
, then python cython_build.py build_ext --inplace
and then python setup.py install
again.
The rationale would be that the C file generated by Cython on my local machine might only be working on Unix like systems. I didn't want to include cython as a dependency, but we might have to.
Hi @dbousque, thanks for the quick response.
Running python setup.py install
indeed results in the same error.
Running pip install -r dev-requirements.py
didn't work first, apparently the listed Cython version (0.29.6) wasn't compatible with my current python version/environment. Now created a new environment with python 3.7.9, there I could install Cython==0.29.6.
But when runnng python cython_build.py build_ext --inplace
I still get about the same error:
(testenv1) C:\Users\nkaen\workspace\new\batch_jaro_winkler\python>python cython_build.py build_ext --inplace
running build_ext
building 'batch_jaro_winkler' extension
creating build\temp.win-amd64-3.7
creating build\temp.win-amd64-3.7\Release
C:\MinGW\bin\gcc.exe -mdll -O -Wall -DMS_WIN64 -IC:\Users\nkaen\Anaconda3\envs\testenv1\include -IC:\Users\nkaen\Anaconda3\envs\testenv1\include -c cbatch_jaro_winkler.c -o build\temp.win-amd64-3.7\Release\cbatch_jaro_winkler.o
In file included from C:\Users\nkaen\Anaconda3\envs\testenv1\include/Python.h:87:0,
from cbatch_jaro_winkler.c:22:
C:\Users\nkaen\Anaconda3\envs\testenv1\include/pytime.h:123:59: warning: 'struct timeval' declared inside parameter list will not be visible outside of this definition or declaration
PyAPI_FUNC(int) _PyTime_FromTimeval(_PyTime_t *tp, struct timeval *tv);
^~~~~~~
C:\Users\nkaen\Anaconda3\envs\testenv1\include/pytime.h:130:12: warning: 'struct timeval' declared inside parameter list will not be visible outside of this definition or declaration
struct timeval *tv,
^~~~~~~
C:\Users\nkaen\Anaconda3\envs\testenv1\include/pytime.h:135:12: warning: 'struct timeval' declared inside parameter list will not be visible outside of this definition or declaration
struct timeval *tv,
^~~~~~~
cbatch_jaro_winkler.c:219:41: warning: division by zero [-Wdiv-by-zero]
enum { __pyx_check_sizeof_voidp = 1 / (int)(SIZEOF_VOID_P == sizeof(void*)) };
^
cbatch_jaro_winkler.c:219:12: error: enumerator value for '__pyx_check_sizeof_voidp' is not an integer constant
enum { __pyx_check_sizeof_voidp = 1 / (int)(SIZEOF_VOID_P == sizeof(void*)) };
^~~~~~~~~~~~~~~~~~~~~~~~
cbatch_jaro_winkler.c:610:36: fatal error: ext/batch_jaro_winkler.h: No such file or directory
#include "ext/batch_jaro_winkler.h"
^
compilation terminated.
error: command 'C:\\MinGW\\bin\\gcc.exe' failed with exit status 1
Let me know how I can help to resolve that issue. Never worked with cython before..
Ok apparently it would be an issue with Cython/MinGW interop: https://github.com/cython/cython/issues/2670#issuecomment-432212671.
Try to run python cython_build.py build_ext --inplace -DMS_WIN64
(with -DMS_WIN64
) then. If that solves it, I guess that you will get a new cbatch_jaro_winkler.c
file that is different from the original one. It would be really nice if you could make a new cython_mingw_build.py
file that is similar to cython_build.py
but has a different target name (cbatch_jaro_winkler.pyx
-> cbatch_jaro_winkler_mingw.pyx
maybe) and have a check in setup.py
to either use cbatch_jaro_winkler.c
or cbatch_jaro_winkler_mingw.c
depending on whether MinGW is used or not. I would have a hard time trying it out myself, I could try to setup an environment similar to yours but that may be painful.
If everything is ok, running python setup.py install
should work correctly. Regarding the missing ext/batch_jaro_winkler.h
header, I forgot to tell you to copy the lib
folder from the root of the git project and paste it as ext
in the python
folder.
Let me know if I can help.
Just tried out python cython_build.py build_ext --inplace -DMS_WIN64
, but still getting the error :(
I've been googling the error as well, and as you did the only response I could find was the -DMS_WIN64 option.. Do you have an idea of something else we could try?
(testenv1) C:\Users\nkaen\workspace\new\batch_jaro_winkler\python>python cython_build.py build_ext --inplace -DMS_WIN64
running build_ext
building 'batch_jaro_winkler' extension
C:\MinGW\bin\gcc.exe -mdll -O -Wall -DMS_WIN64 -DMS_WIN64=1 -I. -IC:\Users\nkaen\Anaconda3\envs\testenv1\include -IC:\Users\nkaen\Anaconda3\envs\testenv1\include -c cbatch_jaro_winkler.c -o build\temp.win-amd64-3.7\Release\cbatch_jaro_winkler.o
In file included from C:\Users\nkaen\Anaconda3\envs\testenv1\include/Python.h:87:0,
from cbatch_jaro_winkler.c:22:
C:\Users\nkaen\Anaconda3\envs\testenv1\include/pytime.h:123:59: warning: 'struct timeval' declared inside parameter list will not be visible outside of this definition or declaration
PyAPI_FUNC(int) _PyTime_FromTimeval(_PyTime_t *tp, struct timeval *tv);
^~~~~~~
C:\Users\nkaen\Anaconda3\envs\testenv1\include/pytime.h:130:12: warning: 'struct timeval' declared inside parameter list will not be visible outside of this definition or declaration
struct timeval *tv,
^~~~~~~
C:\Users\nkaen\Anaconda3\envs\testenv1\include/pytime.h:135:12: warning: 'struct timeval' declared inside parameter list will not be visible outside of this definition or declaration
struct timeval *tv,
^~~~~~~
cbatch_jaro_winkler.c:219:41: warning: division by zero [-Wdiv-by-zero]
enum { __pyx_check_sizeof_voidp = 1 / (int)(SIZEOF_VOID_P == sizeof(void*)) };
^
cbatch_jaro_winkler.c:219:12: error: enumerator value for '__pyx_check_sizeof_voidp' is not an integer constant
enum { __pyx_check_sizeof_voidp = 1 / (int)(SIZEOF_VOID_P == sizeof(void*)) };
^~~~~~~~~~~~~~~~~~~~~~~~
error: command 'C:\\MinGW\\bin\\gcc.exe' failed with exit status 1
Just found this stackoverflow post, which might be relevant: https://stackoverflow.com/questions/55995160/unable-to-convert-cython-generated-c-language-code-to-executable-file
Yep, you could try python cython_build.py build_ext --inplace -DSIZEOF_VOID_P=8 -DMS_WIN64
, it would sound logical that it solves the issue given the error is a complaint that __pyx_check_sizeof_voidp
is not an integer constant.
Already tried both options -DSIZEOF_VOID_P=8 and -DSIZEOF_VOID_P=4, doesn't work either :(
Ah, I just saw that it looks like python cython_build.py
is not trying to recreate the cbatch_jaro_winkler.c
file, you probably need to remove this file. If that doesn't do it, maybe the command line arguments to python cython_build.py
are not passed when doing the extension compilation. In cython_build.py
, maybe try to replace:
ext_modules=cythonize([Extension('batch_jaro_winkler', ['cbatch_jaro_winkler.pyx'])], language_level=python_version)
with:
ext_modules=cythonize([Extension('batch_jaro_winkler', ['cbatch_jaro_winkler.pyx'], define_macros=[('SIZEOF_VOID_P', '8'), ('MS_WIN64', '1')])], language_level=python_version)
Additionally, try to clean the environment before/after each try, like we do in local_build.sh
and cython_build.sh
(remove files and folders mainly). Try to get a build flow as similar as local_build.sh
as possible, maybe adding the C file deletion each time as well.
Just tried that out, unfortunately still getting that same error
I tried to debug for MinGW on my Windows machine but it's a mess. The default Windows C compiler fails for another reason (can't find the size of void *
🤷 ). It looks like building native extensions (maybe Cython extensions in particular) is a mess on Windows: https://wiki.python.org/moin/WindowsCompilers#GCC_-, and I don't think there would be a clean way to support most exotic combinations of Python versions / C compilers (MSVC/mingw32/gcc) / Development environments (like MinGW, virtual envs etc.).
If that works for you, you could go with "Ubuntu for Windows". Or if you really need to get this library working on Windows, things may work well if you go for a standard combination of Python version + MSVC version + standard terminal and PATH setup etc. (whatever that may be) like mentioned in the link from the official CPython website.
I see, thanks a lot for looking into it. For now I will run it in WSL.
Thanks for the report and investigation efforts, have a nice day!
Hi. I have a similar problem on Mac OS.
On Linux Ubuntu the module works perfectly, however when trying to install it on a windows 10 machine with anaconda (python 3.8.5), I get a C compiler error:
I installed the GCC compiler suite on windows using MinGW, following these instructions: https://cython.readthedocs.io/en/latest/src/tutorial/appendix.html
Do you have an idea how we could achieve compability for windows?
Here is the full output i currently get on pip install: