BUGFIX: pystan build on Windows platform using mingw64 gcc

[dxma]

Summary:

Hi, the setup.py script needs a small fix in order to build pystan on Windows 7 x64 platform. Extension("pystan._misc", ["pystan/_misc.pyx"], language='c++', extra_compile_args=extra_compile_args)

Description:

The misc.pyx extension raised an error like below: D:/msys64/mingw64/include/c++/8.3.0/cmath:1121:11: error: '::hypot' has not been declared using ::hypot; ^~~~~

Reason is quite obvious, setup.py doesn't pass extra_compile_args to this extension building block, which contains intented flags for MingW64 platform.

Reproducible Steps:

1. install msys2 64bit and fetch all latest package updates using pacman.
2. install mingw-w64-x86_64-toolchain via pacman.
3. add msys2 mingw64 bin path to PATH environment variable, which in my case is D:\msys64\mingw64\bin
4. alternatively, the first three steps above could be done via following the official documentation page, from the conda environment.
5. install pystan via pip command line.

Current Output:

compile options: '-DMSVCRT_VERSION__=0x1914 -ID:\pyvenv\crushburn\include -IC: \Program Files\Python37\include -IC:\Program Files\Python37\include -c' gcc -g -DDEBUG -DMS_WIN64 -O0 -Wall -Wstrict-prototypes -DMSVCRT_VERSION=0x1 914 -ID:\pyvenv\crushburn\include -IC:\Program Files\Python37\include -IC:\Progr am Files\Python37\include -c pystan/_misc.cpp -o build\temp.win-amd64-3.7\Releas e\pystan_misc.o cc1plus.exe: warning: command line option '-Wstrict-prototypes' is valid for C/O bjC but not for C++ In file included from D:/msys64/mingw64/include/c++/8.3.0/math.h:36, from C:\Program Files\Python37\include/pyport.h:191, from C:\Program Files\Python37\include/Python.h:53, from pystan/_misc.cpp:17: D:/msys64/mingw64/include/c++/8.3.0/cmath:1121:11: error: '::hypot' has not been declared using ::hypot; ^~~~~ error: Command "gcc -g -DDEBUG -DMS_WIN64 -O0 -Wall -Wstrict-prototypes -DMSVC RT_VERSION__=0x1914 -ID:\pyvenv\crushburn\include -IC:\Program Files\Python37\in clude -IC:\Program Files\Python37\include -c pystan/_misc.cpp -o build\temp.win- amd64-3.7\Release\pystan_misc.o" failed with exit status 1

Expected Output:

compile options: '-DMSVCRT_VERSION=0x1914 -ID:\pyvenv\crushburn\include -IC: \Program Files\Python37\include -IC:\Program Files\Python37\include -c' extra options: '-Os -ftemplate-depth-256 -Wno-unused-function -Wno-uninitialized -std=c++11 -D_hypot=hypot -pthread -fexceptions' gcc -g -DDEBUG -DMS_WIN64 -O0 -Wall -Wstrict-prototypes -DMSVCRT_VERSION=0x1 914 -ID:\pyvenv\crushburn\include -IC:\Program Files\Python37\include -IC:\Progr am Files\Python37\include -c pystan/_misc.cpp -o build\temp.win-amd64-3.7\Releas e\pystan_misc.o -Os -ftemplate-depth-256 -Wno-unused-function -Wno-uninitialize d -std=c++11 -D_hypot=hypot -pthread -fexceptions cc1plus.exe: warning: command line option '-Wstrict-prototypes' is valid for C/O bjC but not for C++ g++ -g -shared build\temp.win-amd64-3.7\Release\pystan_misc.o -LD:\pyvenv\crush burn\libs -LC:\Program Files\Python37\libs -LC:\Program Files\Python37 -LD:\pyve nv\crushburn\PCbuild\amd64 -lpython37 -o build\lib.win-amd64-3.7\pystan_misc.cp 37-win_amd64.pyd

PyStan Version:

• 2.18.1.0

Python Version:

• 3.7.0 (official build)

Operating System:

• MSYS_NT-6.1 SQSZ-D0741 2.11.2(0.329/5/3) 2018-11-26 09:22 x86_64 Msys
• Windows 7 x64

[ahartikainen]

Is this when using non-conda mingw-w64?

What does this give you

from Cython.Build.Inline import _get_build_extension
_get_build_extension().compiler

[dxma]

Hi,

1. Yes it is non-conda mingw64 environ, for a reason I choose not to use conda.
2. It is mingw32 compiler as set in distutils config, current cygwin compiler doesn't support my msvc runtime version - 1914.

Cheers,

[ahartikainen]

If _get_build_extension().compiler gives you mingw32, then it should add the correct extra_compile_args for setup.

What does this give you platform.platform()

[ahartikainen]

No, sorry, yes it is missing from misc.pyx. Let's fix it.

[dxma]

Thank you!

• Unfortunately this doesn't work properly while trying the simpliest logistic regression demo. The gcc command did make the dot o file but still returned -1 on command line. That triggered an endless loop of respawn sub-process.
• I also tried building cython using mingw64 toolchain, cython installed but that didn't resolve the problem above.
• An interesting finding from this excersice: pystan setup.py imports numpy.distutils, this triggers a magic to override mingw32 compiler in ccompiler.py. The numpy version bypassed the msvc version check inherited from cygwin compiler defined in vanilla distutils code. It is the major reason that the build succeeded in my environment. Thus I modified pystan/model.py to utilize numpy.distutils.
• Personally I would think the main difference between official windows python + mingw64 and conda windows python + mingw64 (via conda) is the dynamic libpython addition in conda. I guess this library has got rid of the msvc++ linkage.

Final conclusion, for now I will stick with conda python on windows platform. Thanks again for your great work on porting stan to python!

• Dongxu

[dxma]

I should take back the 1st point in previous post:

* Unfortunately this doesn't work properly while trying the simpliest logistic regression demo. The gcc command did make the dot o file but still returned -1 on command line. That triggered an endless loop of respawn sub-process.

This isn't true. The endless respawn is triggered from StanModel.sampling call, which executes in parallel using multiprocessing package. The forked child re-initialize the model object again. Passing n_jobs=1 to sampling() method works around this problem.

[ahartikainen]

Did you try to run the code in if __name__ == "__main__" or in jupyter?

[dxma]

I tried in 2 ways:

1. a python interactive session.
2. wrap the code in test.py and run in conda command line. For some reason it happened only in 2nd.