`npy_cdouble` related error when compiling 1.26.1 with Intel compilers on Windows

[rgommers]

Transferred from https://github.com/numpy/numpy/issues/22711#issuecomment-1785115279 (bug report by @ashish-2022).:

I get this error when compiling numpy 1.26.1 with intel MKL on Windows (Python 3.11.6). is there any extra environment variables I need to set for this?

 [7/489] Compiling C object numpy/core/libnpymath.a.p/meson-generated_ieee754.c.obj
      FAILED: numpy/core/libnpymath.a.p/meson-generated_ieee754.c.obj
      "icl" "-Inumpy\core\libnpymath.a.p" "-Inumpy\core" "-I..\..\numpy\core" "-Inumpy\core\include" "-I..\..\numpy\core\include" "-I..\..\numpy\core\src\npymath" "-I..\..\numpy\core\src\common" "-IC:\DE-Python\Include" "-ID:\Script_VMC\numpy-1.26.1\.mesonpy-okitud_x\build\meson_cpu" "-DNDEBUG" "/MD" "/nologo" "/showIncludes" "/utf-8" "/W2" "/Qstd:c99" "/O3" "/arch:SSE3" "-DNPY_HAVE_SSE2" "-DNPY_HAVE_SSE" "-DNPY_HAVE_SSE3" "-DMS_WIN64=" "/Fdnumpy\core\libnpymath.a.p\meson-generated_ieee754.c.pdb" /Fonumpy/core/libnpymath.a.p/meson-generated_ieee754.c.obj "/c" numpy/core/libnpymath.a.p/ieee754.c
      icl: remark #10441: The Intel(R) C++ Compiler Classic (ICC) is deprecated and will be removed from product release in the second half of 2023. The Intel(R) oneAPI DPC++/C++ Compiler (ICX) is the recommended compiler moving forward. Please transition to use this compiler. Use '/Qdiag-disable:10441' to disable this message.
      ../../numpy/core/include/numpy/npy_common.h(388): catastrophic error: #error directive: npy_cdouble definition is not compatible with C99 complex definition !         Please contact NumPy maintainers and give detailed information about your         compiler and platform
        #error npy_cdouble definition is not compatible with C99 complex definition ! \
         ^

      compilation aborted for numpy/core/libnpymath.a.p/ieee754.c (code 4)

[rgommers]

The relevant code is: https://github.com/numpy/numpy/blob/6207a5203983e720cd36f49a39ba8ea3433c3231/numpy/core/include/numpy/npy_common.h#L387-L392

@ashish-2022 would you mind providing a full build log and the build command you used? Also, this code has changed a lot in the main branch, so if you could check if you are getting the same error there, that would be helpful.

[ashish-2022]

The relevant code is:

https://github.com/numpy/numpy/blob/6207a5203983e720cd36f49a39ba8ea3433c3231/numpy/core/include/numpy/npy_common.h#L387-L392

@ashish-2022 would you mind providing a full build log and the build command you used? Also, this code has changed a lot in the main branch, so if you could check if you are getting the same error there, that would be helpful.

Ok I'll try to build from latest code on GitHUB

[ashish-2022]

Complete build log(code taken from https://files.pythonhosted.org/packages/78/23/f78fd8311e0f710fe1d065d50b92ce0057fe877b8ed7fd41b28ad6865bfc/numpy-1.26.1.tar.gz)

Log Attached. numpy_1_26_1_build_log.txt

Command used to build: python -m pip install .

[ashish-2022]

Code from main branch exited much earlier on my setup at following check:

Has header "complex.h" : YES
      Checking for type "_Fcomplex" : NO

      ..\numpy\_core\meson.build:143:4: ERROR: Problem encountered: "complex.h" header does not include complex type _Fcomplex

      A full log can be found at D:\Test_Numpy\numpy\.mesonpy-gt7tmyk6\meson-logs\meson-log.txt
      [end of output]

Full log here: build_log_main.txt

[lysnikolaou]

Hmm, I assume this happens due to https://github.com/numpy/numpy/blob/cdfbdf428d9df9c7119cecae323512a4cd3f57b7/numpy/_core/meson.build#L127 We should probably special-case Intel there, right?

[ashish-2022]

Hmm, I assume this happens due to

https://github.com/numpy/numpy/blob/cdfbdf428d9df9c7119cecae323512a4cd3f57b7/numpy/_core/meson.build#L127

We should probably special-case Intel there, right?

This can be achieved using compiler.get_id(), this is the list: https://mesonbuild.com/Reference-tables.html#compiler-ids

[lysnikolaou]

@ashish-2022 Can you check whether you can build numpy with #25044?

[ashish-2022]

Hi @lysnikolaou, I had tried just putting that condition in meson.build but we will need more changes related to intel compiler complex types.

[9/491] Compiling C++ object numpy/_core/libnpymath.a.p/src_npymath_halffloat.cpp.obj
      FAILED: numpy/_core/libnpymath.a.p/src_npymath_halffloat.cpp.obj
      "icl" "-Inumpy\_core\libnpymath.a.p" "-Inumpy\_core" "-I..\numpy\_core" "-Inumpy\_core\include" "-I..\numpy\_core\include" "-I..\numpy\_core\src\npymath" "-I..\numpy\_core\src\common" "-IC:\DE-Python\Include" "-ID:\Test_Numpy\numpy\.mesonpy-q6_9ehof\meson_cpu" "-DNDEBUG" "/MD" "/nologo" "/showIncludes" "/utf-8" "/W2" "/EHsc" "/std:c++17" "/permissive-" "/O3" "/arch:SSE3" "-DNPY_HAVE_SSE2" "-DNPY_HAVE_SSE" "-DNPY_HAVE_SSE3" "-DMS_WIN64=" "/Fdnumpy\_core\libnpymath.a.p\src_npymath_halffloat.cpp.pdb" /Fonumpy/_core/libnpymath.a.p/src_npymath_halffloat.cpp.obj "/c" ../numpy/_core/src/npymath/halffloat.cpp
      icl: remark #10441: The Intel(R) C++ Compiler Classic (ICC) is deprecated and will be removed from product release in the second half of 2023. The Intel(R) oneAPI DPC++/C++ Compiler (ICX) is the recommended compiler moving forward. Please transition to use this compiler. Use '/Qdiag-disable:10441' to disable this message.
      C:\Program Files (x86)\Intel\oneAPI\compiler\latest\windows\compiler\include\complex.h(37): warning #1224: #warning directive: "The /Qstd=c99 compilation option is required to enable C99 support for C programs"
                    #warning "The /Qstd=c99 compilation option is required to enable C99 support for C programs"
                     ^

      ../numpy/_core/include/numpy/npy_common.h(397): error: expected a ";"
        typedef double _Complex npy_cdouble;

See full log here: intel_icl_log.txt

Then I also included "/Qstd=c99" flag in compilation but there was no use. need to check with some intel expert @rgommers

[lysnikolaou]

Unfortunately I don't have much experience with Intel and also no Windows machine to reproduce this issue at the moment, so I'll leave this to someone else.

[ashish-2022]

From commits history following members can assign this to right person: @mattip, @chaoyu3

[mattip]

As the intel compiler is proprietary, we would need someone with a proper license to work through this. I do not have such a license.

[seberg]

It looks like at least the old intel compiler doesn't like _Complex in C++ (on windows?). My suspicion would be that needs to be tweaked, half of the branching necessary is already there. Can't we rely on a standard form in C++ mode in all cases?

In either case, it needs someone with the compiler to try out what works well here.

[rgommers]

@mattip that is no longer the case. Well, it's proprietary still, but freely available - you can simple download and install it, no license required.

[rgommers]

The installers are quite cumbersome indeed. You can deselect most components though, so it shouldn't take that much space. That said, this is still a bit of a time sink, and this isn't super urgent probably. I'm planning to get a Windows laptop at the end of the month to be able to deal with this kind of thing more easily. So I can circle back to it then - before 2.0 at least.

[mattip]

Somehow the installer works differently via a remote desktop, it seems to work better with direct access.

[mattip]

The latest oneAPI compiler suite I installed came with icx.exe and not icl.exe. I needed to do this to build:

<install path>\setvars.bat intel64
set CC=icx.exe
set CXX=icx.exe
pip install scipy-openblas64 -r build_requirements.txt -r test_requirements.txt

REM use scipy-openblas wheels to provide OpenBLAS
pip install scipy-openblas64
set PKG_CONFIG_PATH=<path>\.openblas

pip install . --no-build-isolation

On HEAD I get the same error as in the comment above in the compiler check stage. I am starting to work through the needed changes. Many of the warnings come from icx.exe being a clang-based c++ compiler which is used to compile c code.

[ashish-2022]

Hi @mattip, This is how I was building earlier version's of numpy 1.24.3 with intel MKL:

:: ------------------------------------------------------------------------------
:: Compile and install numpy
:: ------------------------------------------------------------------------------
:: Download numpy
powershell Invoke-WebRequest -Uri 'https://files.pythonhosted.org/packages/2c/d4/590ae7df5044465cc9fa2db152ae12468694d62d952b1528ecff328ef7fc/numpy-1.24.3.tar.gz' -OutFile 'numpy-1.24.3.tar.gz'
tar -xvzf numpy-1.24.3.tar.gz
set NUMPY_DIR="numpy-1.24.3"
:: Create site.cfg
echo [mkl] > %NUMPY_DIR%/site.cfg
echo include_dirs = C:\Program Files (x86)\Intel\oneAPI\mkl\latest\include >> %NUMPY_DIR%/site.cfg
echo library_dirs = C:\Program Files (x86)\Intel\oneAPI\mkl\latest\lib\intel64 >>%NUMPY_DIR%/site.cfg
echo libraries = mkl_rt >> %NUMPY_DIR%/site.cfg
:: Start build
python -m pip install --verbose %NUMPY_DIR%/
:: Copy intel MKL libs for numpy
mkdir "%PYTHONHOME%\Lib\site-packages\numpy\.libs"
copy "C:\Program Files (x86)\Intel\oneAPI\mkl\latest\redist\intel64\*.dll" %PYTHONHOME%\Lib\site-packages\numpy\.libs
robocopy "C:\Program Files (x86)\Intel\oneAPI\mkl\latest\redist\intel64\1033" %PYTHONHOME%\Lib\site-packages\numpy\.libs\1033

Please note it required to link to intel MKL libs from site-packages/numpy/.libs/

There was some code I had seen long back for this (dont remember where it is, may be in one of init.py)

[mattip]

Right, but we no longer use site.cfg

[mattip]

There are many warnings from CPython about a missing declaration of struct timeval. This was fixed for CPython 3.12, so I would recommend using that for compilation with the oneAPI compiler (or clang-cl for that matter). See python/cpython#82863 and python/cpython#77532

[mattip]

There are also many warnings about "warning: explicit comparison with NaN in fast floating point mode". It seems meson should use at least -fhonor-infinities -fhonor-nans according to this issue in another repo

[rgommers]

We should be disabling -ffast-math for that on Intel compilers. It's missing in NumPy; I did do it for SciPy:

https://github.com/scipy/scipy/blob/ae026f72fbc3b6c60a5d5b9d2e392b5c1919fd44/meson.build#L91-L117

[mattip]

OK. I will add that to #25044

[eyalk11]

I tried to compile with python3.13t (no GIL) , and got the same error.