MSVC 2019 / MINGW-W64 combo chokes up for build

ilayn commented 5 years ago

Although NumPy compiled succesfully (as far as I can tell) with same toolchain, it started to choke with SciPy and according to Rust community this is because certain headers are inlined apparently (see details below). Hence missing references are caused.

I am not very hopeful since it is Windows but I'd be happy if anybody can help to tame this build chain. Unfortunately, I have no understading of these issues to provide more feedback.

```bash C:\Program Files\mingw-w64\x86_64-7.1.0-posix-seh-rt_v5-rev2\mingw64\bin\gfortran.exe -Wall -g -Wall -g -shared -Wl,--whole-archive ..\..\..\..\..\opt\64\lib\libopenblas.a -Wl,--no-whole-archive -LC:\Program Files\mingw-w64\x86_64-7.1.0-posix-seh-rt_v5-rev2\mingw64\lib\gcc\x86_64-w64-mingw32\7.1.0 -LC:\Users\Ilhan Polat\AppData\Local\Programs\Python\Python36\libs -LC:\Users\Ilhan Polat\AppData\Local\Programs\Python\Python36\PCbuild\amd64 -Lbuild\temp.win-amd64-3.6 -o build\temp.win-amd64-3.6\Release\.libs\liblibopenb.ZN3UGLQCYK32WCJ6CCN3QBEYE5TCWN7J.gfortran-win_amd64.dll -Wl,--allow-multiple-definition -Wl,--output-def,build\temp.win-amd64-3.6\Release\liblibopenb.ZN3UGLQCYK32WCJ6CCN3QBEYE5TCWN7J.gfortran-win_amd64.def -Wl,--export-all-symbols -Wl,--enable-auto-import -static -mlong-double-64 ..\..\..\..\..\opt\64\lib\libopenblas.a(cblas_xerbla.obj):xerbla.c:(.text+0xf): undefined reference to `__imp___acrt_iob_func' ..\..\..\..\..\opt\64\lib\libopenblas.a(strmv_NUU.obj):trmv_U.c:(.text+0x17e): undefined reference to `__imp___acrt_iob_func' ..\..\..\..\..\opt\64\lib\libopenblas.a(strmv_NUN.obj):trmv_U.c:(.text+0x1c7): undefined reference to `__imp___acrt_iob_func' ..\..\..\..\..\opt\64\lib\libopenblas.a(dtrmv_NUU.obj):trmv_U.c:(.text+0x17e): undefined reference to `__imp___acrt_iob_func' ..\..\..\..\..\opt\64\lib\libopenblas.a(dtrmv_NUN.obj):trmv_U.c:(.text+0x1c7): undefined reference to `__imp___acrt_iob_func' ..\..\..\..\..\opt\64\lib\libopenblas.a(openblas_error_handle.obj):openblas_error_handle.c:(.text+0x24): more undefined references to `__imp___acrt_iob_func' follow collect2.exe: error: ld returned 1 exit status error: Command "C:\Program Files\mingw-w64\x86_64-7.1.0-posix-seh-rt_v5-rev2\mingw64\bin\gfortran.exe -Wall -g -Wall -g -shared -Wl,--whole-archive ..\..\..\..\..\opt\64\lib\libopenblas.a -Wl,--no-whole-archive -LC:\Program Files\mingw-w64\x86_64-7.1.0-posix-seh-rt_v5-rev2\mingw64\lib\gcc\x86_64-w64-mingw32\7.1.0 -LC:\Users\Ilhan Polat\AppData\Local\Programs\Python\Python36\libs -LC:\Users\Ilhan Polat\AppData\Local\Programs\Python\Python36\PCbuild\amd64 -Lbuild\temp.win-amd64-3.6 -o build\temp.win-amd64-3.6\Release\.libs\liblibopenb.ZN3UGLQCYK32WCJ6CCN3QBEYE5TCWN7J.gfortran-win_amd64.dll -Wl,--allow-multiple-definition -Wl,--output-def,build\temp.win-amd64-3.6\Release\liblibopenb.ZN3UGLQCYK32WCJ6CCN3QBEYE5TCWN7J.gfortran-win_amd64.def -Wl,--export-all-symbols -Wl,--enable-auto-import -static -mlong-double-64" failed with exit status 1 ```

pv commented 5 years ago

Up to and including that point, it should be pure GCC toolchain, there's no MSVC involved yet. Maybe the libopenblas.a (where did you get it from?) you have comes from incompatible mingw version, than what you are building scipy with?

tylerjereddy commented 5 years ago

Looking over our CI, I think mingw 6.3.0 or 6.4.0 may be a better match for providing the runtime libs needed by OpenBLAS (libquadmath & so on), but as Pauli says would be useful to know where you are getting that OpenBLAS..

ilayn commented 5 years ago

I directly downloaded the precompiled binaries for windows from the sourceforge repository version 0.3.7 and setup a site.cfg accordingly for numpy. Same didn't go through for SciPy.

pv commented 5 years ago

The openblas SF package contains a prebuilt MSVC .lib library, so numpy probably didn't use mingw at all. gfortran however needs to use the .a library, which presumably was built with a mingw version incompatible with what you have.

pv commented 5 years ago

ilayn commented 5 years ago

See also https://scipy.github.io/devdocs/building/windows.html

Indeed, I was trying to fix those instructions since I was one of the authors back then. But currently it is completely out of date and getting some traffic.

OK brought down to a single instance of the error given above so I think I can provide more information since I'm a bit more aware of the tools I'm using.

What I have been doing is the following for building the OpenBLAS

Clone and switch to the correct release (v0.3.7 as of writing)
Then in a MSYS2 Ming-w64 shell I've installed the compilers cmake perl
then fire up the build process via make BINARY=64 CC=gcc FC=gfortran

The details of compilers are

```bash $ gcc -v Using built-in specs. COLLECT_GCC=C:\msys64\mingw64\bin\gcc.exe COLLECT_LTO_WRAPPER=C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/9.2.0/lto-wrapper.exe Target: x86_64-w64-mingw32 Configured with: ../gcc-9.2.0/configure --prefix=/mingw64 --with-local-prefix=/mingw64/local --build=x86_64-w64-mingw32 --host=x86_64-w64-mingw32 --target=x86_64-w64-mingw32 --with-native-system-header-dir=/mingw64/x86_64-w64-mingw32/include --libexecdir=/mingw64/lib --enable-bootstrap --with-arch=x86-64 --with-tune=generic --enable-languages=c,lto,c++,fortran,ada,objc,obj-c++ --enable-shared --enable-static --enable-libatomic --enable-threads=posix --enable-graphite --enable-fully-dynamic-string --enable-libstdcxx-filesystem-ts=yes --enable-libstdcxx-time=yes --disable-libstdcxx-pch --disable-libstdcxx-debug --disable-isl-version-check --enable-lto --enable-libgomp --disable-multilib --enable-checking=release --disable-rpath --disable-win32-registry --disable-nls --disable-werror --disable-symvers --enable-plugin --with-libiconv --with-system-zlib --with-gmp=/mingw64 --with-mpfr=/mingw64 --with-mpc=/mingw64 --with-isl=/mingw64 --with-pkgversion='Rev1, Built by MSYS2 project' --with-bugurl=https://sourceforge.net/projects/msys2 --with-gnu-as --with-gnu-ld Thread model: posix gcc version 9.2.0 (Rev1, Built by MSYS2 project) ```

and for gfortran

```bash $ gfortran -v Using built-in specs. COLLECT_GCC=C:\msys64\mingw64\bin\gfortran.exe COLLECT_LTO_WRAPPER=C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/9.2.0/lto-wrapper.exe Target: x86_64-w64-mingw32 Configured with: ../gcc-9.2.0/configure --prefix=/mingw64 --with-local-prefix=/mingw64/local --build=x86_64-w64-mingw32 --host=x86_64-w64-mingw32 --target=x86_64-w64-mingw32 --with-native-system-header-dir=/mingw64/x86_64-w64-mingw32/include --libexecdir=/mingw64/lib --enable-bootstrap --with-arch=x86-64 --with-tune=generic --enable-languages=c,lto,c++,fortran,ada,objc,obj-c++ --enable-shared --enable-static --enable-libatomic --enable-threads=posix --enable-graphite --enable-fully-dynamic-string --enable-libstdcxx-filesystem-ts=yes --enable-libstdcxx-time=yes --disable-libstdcxx-pch --disable-libstdcxx-debug --disable-isl-version-check --enable-lto --enable-libgomp --disable-multilib --enable-checking=release --disable-rpath --disable-win32-registry --disable-nls --disable-werror --disable-symvers --enable-plugin --with-libiconv --with-system-zlib --with-gmp=/mingw64 --with-mpfr=/mingw64 --with-mpc=/mingw64 --with-isl=/mingw64 --with-pkgversion='Rev1, Built by MSYS2 project' --with-bugurl=https://sourceforge.net/projects/msys2 --with-gnu-as --with-gnu-ld Thread model: posix gcc version 9.2.0 (Rev1, Built by MSYS2 project) ```

When the build is finished I also use make install PREFIX=/c/opt to get the artifacts

After that I turn to a regular windows command prompt and then first add a site.cfg

echo "[lapack_src]
>> libraries = openblas
>> library_dirs = C:\opt\lib
>> include_dirs = C:\opt\include
>> runtime_library_dirs = C:\opt\lib" > site.cfg

This time python setup.py build fails with not being able to find the openblas libs

```bash ~\Documents\GitHub\scipy [master ≡]> python .\setup.py build Cythonizing sources Running scipy\linalg\_generate_pyx.py Running scipy\special\_generate_pyx.py scipy/linalg/_generate_pyx.py: all files up-to-date scipy/special/_generate_pyx.py: all files up-to-date scipy\cluster\_hierarchy.pyx has not changed scipy\io\matlab\mio_utils.pyx has not changed scipy\io\matlab\streams.pyx has not changed scipy\linalg\cython_blas.pyx has not changed scipy\linalg\cython_lapack.pyx has not changed scipy\interpolate\_bspl.pyx has not changed scipy\interpolate\_ppoly.pyx has not changed scipy\ndimage\src\_cytest.pyx has not changed scipy\cluster\_optimal_leaf_ordering.pyx has not changed scipy\optimize\_bglu_dense.pyx has not changed scipy\optimize\_group_columns.pyx has not changed scipy\optimize\cython_optimize\_zeros.pyx.in has not changed scipy\linalg\_decomp_update.pyx.in has not changed scipy\linalg\_solve_toeplitz.pyx has not changed scipy\io\matlab\mio5_utils.pyx has not changed scipy\ndimage\src\_ni_label.pyx has not changed scipy\signal\_spectral.pyx has not changed scipy\signal\_upfirdn_apply.pyx has not changed scipy\sparse\_csparsetools.pyx.in has not changed scipy\sparse\csgraph\_min_spanning_tree.pyx has not changed scipy\sparse\csgraph\_reordering.pyx has not changed scipy\sparse\csgraph\_shortest_path.pyx has not changed scipy\sparse\csgraph\_tools.pyx has not changed scipy\cluster\_vq.pyx has not changed scipy\spatial\ckdtree.pyx has not changed scipy\spatial\qhull.pyx has not changed scipy\optimize\_lsq\givens_elimination.pyx has not changed scipy\spatial\_voronoi.pyx has not changed scipy\special\cython_special.pyx has not changed scipy\signal\_peak_finding_utils.pyx has not changed scipy\special\_ellip_harm_2.pyx has not changed scipy\special\_test_round.pyx has not changed scipy\special\_ufuncs.pyx has not changed scipy\spatial\_hausdorff.pyx has not changed scipy\stats\_stats.pyx has not changed scipy\_lib\messagestream.pyx has not changed scipy\_lib\_ccallback_c.pyx has not changed scipy\sparse\csgraph\_traversal.pyx has not changed scipy\fftpack\convolve.pyx has not changed scipy\interpolate\interpnd.pyx has not changed scipy\special\_ufuncs_cxx.pyx has not changed scipy\optimize\_trlib\_trlib.pyx has not changed scipy\signal\_max_len_seq_inner.pyx has not changed scipy\special\_comb.pyx has not changed Running from scipy source directory. lapack_opt_info: lapack_mkl_info: No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils customize MSVCCompiler libraries mkl_rt not found in ['C:\\Users\\Ilhan Polat\\AppData\\Local\\Programs\\Python\\Python36\\lib', 'C:\\', 'C:\\Users\\Ilhan Polat\\AppData\\Local\\Programs\\Python\\Python36\\libs'] NOT AVAILABLE openblas_lapack_info: No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils customize MSVCCompiler No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils customize MSVCCompiler libraries openblas not found in ['C:\\Users\\Ilhan Polat\\AppData\\Local\\Programs\\Python\\Python36\\lib', 'C:\\', 'C:\\Users\\Ilhan Polat\\AppData\\Local\\Programs\\Python\\Python36\\libs'] get_default_fcompiler: matching types: '['gnu', 'intelv', 'absoft', 'compaqv', 'intelev', 'gnu95', 'g95', 'intelvem', 'intelem', 'flang']' customize GnuFCompiler Could not locate executable g77 Could not locate executable f77 customize IntelVisualFCompiler Could not locate executable ifort Could not locate executable ifl customize AbsoftFCompiler Could not locate executable f90 customize CompaqVisualFCompiler Could not locate executable DF customize IntelItaniumVisualFCompiler Could not locate executable efl customize Gnu95FCompiler Found executable C:\Program Files\mingw-w64\x86_64-7.1.0-posix-seh-rt_v5-rev2\mingw64\bin\gfortran.exe Using built-in specs. COLLECT_GCC=C:\Program Files\mingw-w64\x86_64-7.1.0-posix-seh-rt_v5-rev2\mingw64\bin\gfortran.exe COLLECT_LTO_WRAPPER=C:/Program\ Files/mingw-w64/x86_64-7.1.0-posix-seh-rt_v5-rev2/mingw64/bin/../libexec/gcc/x86_64-w64-mingw32/7.1.0/lto-wrapper.exe Target: x86_64-w64-mingw32 Configured with: ../../../src/gcc-7.1.0/configure --host=x86_64-w64-mingw32 --build=x86_64-w64-mingw32 --target=x86_64-w64-mingw32 --prefix=/mingw64 --with-sysroot=/c/mingw710/x86_64-710-posix-seh-rt_v5-rev2/mingw64 --enable-shared --enable-static --disable-multilib --enable-languages=c,c++,fortran,lto --enable-libstdcxx-time=yes --enable-threads=posix --enable-libgomp --enable-libatomic --enable-lto --enable-graphite --enable-checking=release --enable-fully-dynamic-string --enable-version-specific-runtime-libs --enable-libstdcxx-filesystem-ts=yes --disable-libstdcxx-pch --disable-libstdcxx-debug --enable-bootstrap --disable-rpath --disable-win32-registry --disable-nls --disable-werror --disable-symvers --with-gnu-as --with-gnu-ld --with-arch=nocona --with-tune=core2 --with-libiconv --with-system-zlib --with-gmp=/c/mingw710/prerequisites/x86_64-w64-mingw32-static --with-mpfr=/c/mingw710/prerequisites/x86_64-w64-mingw32-static --with-mpc=/c/mingw710/prerequisites/x86_64-w64-mingw32-static --with-isl=/c/mingw710/prerequisites/x86_64-w64-mingw32-static --with-pkgversion='x86_64-posix-seh-rev2, Built by MinGW-W64 project' --with-bugurl=https://sourceforge.net/projects/mingw-w64 CFLAGS='-O2 -pipe -fno-ident -I/c/mingw710/x86_64-710-posix-seh-rt_v5-rev2/mingw64/opt/include -I/c/mingw710/prerequisites/x86_64-zlib-static/include -I/c/mingw710/prerequisites/x86_64-w64-mingw32-static/include' CXXFLAGS='-O2 -pipe -fno-ident -I/c/mingw710/x86_64-710-posix-seh-rt_v5-rev2/mingw64/opt/include -I/c/mingw710/prerequisites/x86_64-zlib-static/include -I/c/mingw710/prerequisites/x86_64-w64-mingw32-static/include' CPPFLAGS=' -I/c/mingw710/x86_64-710-posix-seh-rt_v5-rev2/mingw64/opt/include -I/c/mingw710/prerequisites/x86_64-zlib-static/include -I/c/mingw710/prerequisites/x86_64-w64-mingw32-static/include' LDFLAGS='-pipe -fno-ident -L/c/mingw710/x86_64-710-posix-seh-rt_v5-rev2/mingw64/opt/lib -L/c/mingw710/prerequisites/x86_64-zlib-static/lib -L/c/mingw710/prerequisites/x86_64-w64-mingw32-static/lib ' Thread model: posix gcc version 7.1.0 (x86_64-posix-seh-rev2, Built by MinGW-W64 project) NOT AVAILABLE openblas_clapack_info: No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils customize MSVCCompiler No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils customize MSVCCompiler libraries openblas,lapack not found in ['C:\\Users\\Ilhan Polat\\AppData\\Local\\Programs\\Python\\Python36\\lib', 'C:\\', 'C:\\Users\\Ilhan Polat\\AppData\\Local\\Programs\\Python\\Python36\\libs'] get_default_fcompiler: matching types: '['gnu', 'intelv', 'absoft', 'compaqv', 'intelev', 'gnu95', 'g95', 'intelvem', 'intelem', 'flang']' customize GnuFCompiler customize IntelVisualFCompiler customize AbsoftFCompiler customize CompaqVisualFCompiler customize IntelItaniumVisualFCompiler customize Gnu95FCompiler Using built-in specs. COLLECT_GCC=C:\Program Files\mingw-w64\x86_64-7.1.0-posix-seh-rt_v5-rev2\mingw64\bin\gfortran.exe COLLECT_LTO_WRAPPER=C:/Program\ Files/mingw-w64/x86_64-7.1.0-posix-seh-rt_v5-rev2/mingw64/bin/../libexec/gcc/x86_64-w64-mingw32/7.1.0/lto-wrapper.exe Target: x86_64-w64-mingw32 Configured with: ../../../src/gcc-7.1.0/configure --host=x86_64-w64-mingw32 --build=x86_64-w64-mingw32 --target=x86_64-w64-mingw32 --prefix=/mingw64 --with-sysroot=/c/mingw710/x86_64-710-posix-seh-rt_v5-rev2/mingw64 --enable-shared --enable-static --disable-multilib --enable-languages=c,c++,fortran,lto --enable-libstdcxx-time=yes --enable-threads=posix --enable-libgomp --enable-libatomic --enable-lto --enable-graphite --enable-checking=release --enable-fully-dynamic-string --enable-version-specific-runtime-libs --enable-libstdcxx-filesystem-ts=yes --disable-libstdcxx-pch --disable-libstdcxx-debug --enable-bootstrap --disable-rpath --disable-win32-registry --disable-nls --disable-werror --disable-symvers --with-gnu-as --with-gnu-ld --with-arch=nocona --with-tune=core2 --with-libiconv --with-system-zlib --with-gmp=/c/mingw710/prerequisites/x86_64-w64-mingw32-static --with-mpfr=/c/mingw710/prerequisites/x86_64-w64-mingw32-static --with-mpc=/c/mingw710/prerequisites/x86_64-w64-mingw32-static --with-isl=/c/mingw710/prerequisites/x86_64-w64-mingw32-static --with-pkgversion='x86_64-posix-seh-rev2, Built by MinGW-W64 project' --with-bugurl=https://sourceforge.net/projects/mingw-w64 CFLAGS='-O2 -pipe -fno-ident -I/c/mingw710/x86_64-710-posix-seh-rt_v5-rev2/mingw64/opt/include -I/c/mingw710/prerequisites/x86_64-zlib-static/include -I/c/mingw710/prerequisites/x86_64-w64-mingw32-static/include' CXXFLAGS='-O2 -pipe -fno-ident -I/c/mingw710/x86_64-710-posix-seh-rt_v5-rev2/mingw64/opt/include -I/c/mingw710/prerequisites/x86_64-zlib-static/include -I/c/mingw710/prerequisites/x86_64-w64-mingw32-static/include' CPPFLAGS=' -I/c/mingw710/x86_64-710-posix-seh-rt_v5-rev2/mingw64/opt/include -I/c/mingw710/prerequisites/x86_64-zlib-static/include -I/c/mingw710/prerequisites/x86_64-w64-mingw32-static/include' LDFLAGS='-pipe -fno-ident -L/c/mingw710/x86_64-710-posix-seh-rt_v5-rev2/mingw64/opt/lib -L/c/mingw710/prerequisites/x86_64-zlib-static/lib -L/c/mingw710/prerequisites/x86_64-w64-mingw32-static/lib ' Thread model: posix gcc version 7.1.0 (x86_64-posix-seh-rev2, Built by MinGW-W64 project) NOT AVAILABLE flame_info: No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils customize MSVCCompiler libraries flame not found in ['C:\\Users\\Ilhan Polat\\AppData\\Local\\Programs\\Python\\Python36\\lib', 'C:\\', 'C:\\Users\\Ilhan Polat\\AppData\\Local\\Programs\\Python\\Python36\\libs'] NOT AVAILABLE atlas_3_10_threads_info: Setting PTATLAS=ATLAS No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils customize MSVCCompiler libraries lapack_atlas not found in C:\Users\Ilhan Polat\AppData\Local\Programs\Python\Python36\lib No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils customize MSVCCompiler libraries tatlas,tatlas not found in C:\Users\Ilhan Polat\AppData\Local\Programs\Python\Python36\lib No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils customize MSVCCompiler libraries lapack_atlas not found in C:\ No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils customize MSVCCompiler libraries tatlas,tatlas not found in C:\ No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils customize MSVCCompiler libraries lapack_atlas not found in C:\Users\Ilhan Polat\AppData\Local\Programs\Python\Python36\libs No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils customize MSVCCompiler libraries tatlas,tatlas not found in C:\Users\Ilhan Polat\AppData\Local\Programs\Python\Python36\libs NOT AVAILABLE atlas_3_10_info: No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils customize MSVCCompiler libraries lapack_atlas not found in C:\Users\Ilhan Polat\AppData\Local\Programs\Python\Python36\lib No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils customize MSVCCompiler libraries satlas,satlas not found in C:\Users\Ilhan Polat\AppData\Local\Programs\Python\Python36\lib No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils customize MSVCCompiler libraries lapack_atlas not found in C:\ No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils customize MSVCCompiler libraries satlas,satlas not found in C:\ No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils customize MSVCCompiler libraries lapack_atlas not found in C:\Users\Ilhan Polat\AppData\Local\Programs\Python\Python36\libs No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils customize MSVCCompiler libraries satlas,satlas not found in C:\Users\Ilhan Polat\AppData\Local\Programs\Python\Python36\libs NOT AVAILABLE atlas_threads_info: Setting PTATLAS=ATLAS No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils customize MSVCCompiler libraries lapack_atlas not found in C:\Users\Ilhan Polat\AppData\Local\Programs\Python\Python36\lib No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils customize MSVCCompiler libraries ptf77blas,ptcblas,atlas not found in C:\Users\Ilhan Polat\AppData\Local\Programs\Python\Python36\lib No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils customize MSVCCompiler libraries lapack_atlas not found in C:\ No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils customize MSVCCompiler libraries ptf77blas,ptcblas,atlas not found in C:\ No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils customize MSVCCompiler libraries lapack_atlas not found in C:\Users\Ilhan Polat\AppData\Local\Programs\Python\Python36\libs No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils customize MSVCCompiler libraries ptf77blas,ptcblas,atlas not found in C:\Users\Ilhan Polat\AppData\Local\Programs\Python\Python36\libs NOT AVAILABLE atlas_info: No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils customize MSVCCompiler libraries lapack_atlas not found in C:\Users\Ilhan Polat\AppData\Local\Programs\Python\Python36\lib No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils customize MSVCCompiler libraries f77blas,cblas,atlas not found in C:\Users\Ilhan Polat\AppData\Local\Programs\Python\Python36\lib No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils customize MSVCCompiler libraries lapack_atlas not found in C:\ No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils customize MSVCCompiler libraries f77blas,cblas,atlas not found in C:\ No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils customize MSVCCompiler libraries lapack_atlas not found in C:\Users\Ilhan Polat\AppData\Local\Programs\Python\Python36\libs No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils customize MSVCCompiler libraries f77blas,cblas,atlas not found in C:\Users\Ilhan Polat\AppData\Local\Programs\Python\Python36\libs NOT AVAILABLE accelerate_info: NOT AVAILABLE lapack_info: No module named 'numpy.distutils._msvccompiler' in numpy.distutils; trying from distutils customize MSVCCompiler libraries lapack not found in ['C:\\Users\\Ilhan Polat\\AppData\\Local\\Programs\\Python\\Python36\\lib', 'C:\\', 'C:\\Users\\Ilhan Polat\\AppData\\Local\\Programs\\Python\\Python36\\libs'] NOT AVAILABLE C:\Users\Ilhan Polat\AppData\Local\Programs\Python\Python36\lib\site-packages\numpy-1.18.0.dev0+77b421d-py3.6-win-amd64.egg\numpy\distutils\system_info.py:1712: UserWarning: Lapack (http://www.netlib.org/lapack/) libraries not found. Directories to search for the libraries can be specified in the numpy/distutils/site.cfg file (section [lapack]) or by setting the LAPACK environment variable. if getattr(self, '_calc_info_{}'.format(lapack))(): lapack_src_info: NOT AVAILABLE C:\Users\Ilhan Polat\AppData\Local\Programs\Python\Python36\lib\site-packages\numpy-1.18.0.dev0+77b421d-py3.6-win-amd64.egg\numpy\distutils\system_info.py:1712: UserWarning: Lapack (http://www.netlib.org/lapack/) sources not found. Directories to search for the sources can be specified in the numpy/distutils/site.cfg file (section [lapack_src]) or by setting the LAPACK_SRC environment variable. if getattr(self, '_calc_info_{}'.format(lapack))(): NOT AVAILABLE Traceback (most recent call last): File ".\setup.py", line 540, in setup_package() File ".\setup.py", line 536, in setup_package setup(**metadata) File "C:\Users\Ilhan Polat\AppData\Local\Programs\Python\Python36\lib\site-packages\numpy-1.18.0.dev0+77b421d-py3.6-win-amd64.egg\numpy\distutils\core.py", line 137, in setup config = configuration() File ".\setup.py", line 435, in configuration raise NotFoundError(msg) numpy.distutils.system_info.NotFoundError: No lapack/blas resources found. ```

So back to our previous instructions (deleting the site.cfg file and placing/renaming the .a file in the ../../programs/libs/openblas.a folder). Now from the build logs I can see that linalg.lapack module and other fortran parts are compiled and comes down to the scipy.cluster and exits with the following

creating build\temp.win-amd64-3.6\Release\scipy
creating build\temp.win-amd64-3.6\Release\scipy\cluster
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.22.27905\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MT -DHAVE_CBLAS -IC:\Users\Ilhan Polat\AppData\Local\Programs\Python\Python36\lib\site-packages\numpy-1.18.0.dev0+77b421d-py3.6-win-amd64.egg\numpy\core\include -IC:\Users\Ilhan Polat\AppData\Local\Programs\Python\Python36\lib\site-packages\numpy-1.18.0.dev0+77b421d-py3.6-win-amd64.egg\numpy\core\include -IC:\Users\Ilhan Polat\AppData\Local\Programs\Python\Python36\include -IC:\Users\Ilhan Polat\AppData\Local\Programs\Python\Python36\include -IC:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.22.27905\include -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt -IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt /Tcscipy\cluster\_vq.c /Fobuild\temp.win-amd64-3.6\Release\scipy\cluster\_vq.obj
could not find library 'openblas' in directories ['C:\\Users\\Ilhan Polat\\Documents\\GitHub\\scipy\\build\\openblas']
C:\Program Files\mingw-w64\x86_64-7.1.0-posix-seh-rt_v5-rev2\mingw64\bin\gfortran.exe -Wall -g -Wall -g -shared -Wl,--whole-archive ..\..\..\AppData\Local\Programs\Python\Python36\lib\openblas.a -Wl,--no-whole-archive -LC:\Program Files\mingw-w64\x86_64-7.1.0-posix-seh-rt_v5-rev2\mingw64\lib\gcc\x86_64-w64-mingw32\7.1.0 -LC:\Users\Ilhan Polat\AppData\Local\Programs\Python\Python36\libs -LC:\Users\Ilhan Polat\AppData\Local\Programs\Python\Python36\PCbuild\amd64 -Lbuild\temp.win-amd64-3.6 -o build\temp.win-amd64-3.6\Release\.libs\libopenblas.5QHOTOUO4U4ER4LE7BFKCC7L4PACMGUD.gfortran-win_amd64.dll -Wl,--allow-multiple-definition -Wl,--output-def,build\temp.win-amd64-3.6\Release\libopenblas.5QHOTOUO4U4ER4LE7BFKCC7L4PACMGUD.gfortran-win_amd64.def -Wl,--export-all-symbols -Wl,--enable-auto-import -static -mlong-double-64
..\..\..\AppData\Local\Programs\Python\Python36\lib\openblas.a(cblas_xerbla.obj):xerbla.c:(.text+0xc): undefined reference to `__imp___acrt_iob_func'
..\..\..\AppData\Local\Programs\Python\Python36\lib\openblas.a(openblas_error_handle.obj):openblas_error_handle.c:(.text+0x27): undefined reference to `__imp___acrt_iob_func'
collect2.exe: error: ld returned 1 exit status
error: Command "C:\Program Files\mingw-w64\x86_64-7.1.0-posix-seh-rt_v5-rev2\mingw64\bin\gfortran.exe -Wall -g -Wall -g -shared -Wl,--whole-archive ..\..\..\AppData\Local\Programs\Python\Python36\lib\openblas.a -Wl,--no-whole-archive -LC:\Program Files\mingw-w64\x86_64-7.1.0-posix-seh-rt_v5-rev2\mingw64\lib\gcc\x86_64-w64-mingw32\7.1.0 -LC:\Users\Ilhan Polat\AppData\Local\Programs\Python\Python36\libs -LC:\Users\Ilhan Polat\AppData\Local\Programs\Python\Python36\PCbuild\amd64 -Lbuild\temp.win-amd64-3.6 -o build\temp.win-amd64-3.6\Release\.libs\libopenblas.5QHOTOUO4U4ER4LE7BFKCC7L4PACMGUD.gfortran-win_amd64.dll -Wl,--allow-multiple-definition -Wl,--output-def,build\temp.win-amd64-3.6\Release\libopenblas.5QHOTOUO4U4ER4LE7BFKCC7L4PACMGUD.gfortran-win_amd64.def -Wl,--export-all-symbols -Wl,--enable-auto-import -static -mlong-double-64" failed with exit status 1

Any pointers would be appreciated.

ilayn commented 5 years ago

This bit from a SO answer looks relevant too

pv commented 5 years ago

I would suggest to try to build an openblas.a that can be successfully built to a DLL, using the command the build in the end fails with.

Note especially that it's trying to use gfortran from C:\Program Files\mingw-w64\x86_64-7.1.0-posix-seh-rt_v5-rev2\mingw64\bin\gfortran.exe whereas you appear to have built openblas with a different version of gcc/gfortran (9.2.0). Try using just one version of mingw, to avoid the runtime mismatch problems I mentioned earlier.

pv commented 5 years ago

Anyway good luck. To my understanding the windows build instructions using msys2 and msvc on the web page should still be applicable. Indeed, the scipy-wheels repository is successfully building wheels every week on appveyor with a similar setup.

ilayn commented 5 years ago

I think you are right those 7.1.0 items are probably from an early installation ended up on the PATH env var.

To my understanding the windows build instructions using msys2 and msvc on the web page should still be applicable. Indeed, the scipy-wheels repository is successfully building wheels every week on appveyor with a similar setup.

Yes should be but the repo is retired (#9669) so I'm trying to make an overhaul directly building from sources without so much extras.

pv commented 5 years ago

I see, you mean the discussion about the openblas build is out of date. I haven't checked that part.

ilayn commented 5 years ago

Indeed, and straight from the Sourceforge libs didn't end up well so I was trying to build-your-own-openblas path.

ilayn commented 5 years ago

Thanks @pv for noticing that version detail. It was indeed the culprit. The build was choosing mingw64 compilers instead of the msys2 ones which have been used to build the openblas lib.

The fix is to make sure that the compilers used to build OpenBLAS has highest priority in the PATH environment variables.

I can finally start working on that Windows build instructions page.

mdhaber commented 5 years ago

@ilayn that would be great.

Note @timmclennan and I spent most of the SciPy Conference sprint trying to build on Windows but didn't get it working.

That's part of the reason for the updates to the Linux Quickstart (to mention the possibility of using WSL or EC2 instances from a Windows machine) and the new Docker Quickstart (which work on Windows). Still, a native Windows Quickstart would be very helpful. If you are going to write something, please consider starting from one of the Quickstarts and adding it to the newer Development Environment section of the contributor guide rather than revising the older Building from sources section. Sure, we might move the Quickstarts to their own page like the Building instructions, but once we have a replacement for each of the Building instructions I'd like to remove all that material as it's somewhat outdated.

Actually, if you get it working, maybe we can set up a Windows-based Docker container so it is very repeatable. I really like to be able to build for Windows in a container, then I could do most of my work outside of it.

ilayn commented 5 years ago

Yes I think I've brought it to a base minimum set of instructions and made it working, quickly wrapping the ?geequ and ?geepub as we speak.

Do you want to get rid of the Building from sources section eventually?

mdhaber commented 5 years ago

Yes

ilayn commented 5 years ago

OK. Let me try to cheat from the recent docker efforts. Fingers crossed 😃

timmclennan commented 5 years ago

Yeah I managed to get scipy building using MSVC 2015 and the Intel Fortran compiler. It isn't actually too hard if you effectively follow the conda feedstock recipe. The biggest issues I ran into were a couple of tests hanging.

After the conference I focused on figure out what the issue was with the tests. It turns out that these particular tests all rely on the multiprocessing.Pool functionality to run a multiprocess map operation, which fails under the scipy integrated test environment.

The failure seems unrelated to scipy and appears to related to running pytests that change the directory, use the multiprocess pool functionality and the windows MKL version of numpy. Under those circumstances a failure case can be found that has nothing directly to do with scipy. One final point is that the conda builds disable the relevant tests.

At some point I intended to write this up further but have been pretty slammed for work in the time since.

scipy / scipy

MSVC 2019 / MINGW-W64 combo chokes up for build #10696