search

serge-sans-paille / pythran

Ahead of Time compiler for numeric kernels
https://pythran.readthedocs.io
BSD 3-Clause "New" or "Revised" License
2.01k stars 193 forks source link

The compilation fails on Apple M1 #1925

Closed certik closed 3 years ago

certik commented 3 years ago

Using the example from the tutorial using pythran 0.10.0 from conda-forge on Apple M1:

def dprod(arr0, arr1):
    return sum([x*y for x,y in zip(arr0, arr1)])

I get:

$ pythran dprod.py   
WARNING:root:No pythran specification, nothing will be exported
WARNING: Compilation error, trying hard to find its origin...
WARNING:pythran:Compilation error, trying hard to find its origin...
WARNING: Nop, I'm going to flood you with C++ errors!
WARNING:pythran:Nop, I'm going to flood you with C++ errors!
CRITICAL: Cover me Jack. Jack? Jaaaaack!!!!
E: error: Command "clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O2 -Wall -fPIC -O2 -isystem /Users/certik/miniforge3/envs/pythran/include -arch arm64 -fPIC -O2 -isystem /Users/certik/miniforge3/envs/pythran/include -arch arm64 -DENABLE_PYTHON_MODULE -D__PYTHRAN__=3 -DPYTHRAN_BLAS_BLAS -I/Users/certik/miniforge3/envs/pythran/lib/python3.10/site-packages/pythran -I/Users/certik/miniforge3/envs/pythran/lib/python3.10/site-packages/numpy/core/include -I/Users/certik/miniforge3/envs/pythran/include -I/Users/certik/miniforge3/envs/pythran/include/python3.10 -c /var/folders/43/nspmtcpj5r589x58n_spl6jm000fsc/T/tmp0ikdp89m.cpp -o /var/folders/43/nspmtcpj5r589x58n_spl6jm000fsc/T/tmpra07avzl/var/folders/43/nspmtcpj5r589x58n_spl6jm000fsc/T/tmp0ikdp89m.o -std=c++11 -fno-math-errno -Wno-unused-function" failed with exit status 1
CRITICAL:pythran:Cover me Jack. Jack? Jaaaaack!!!!
E: error: Command "clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O2 -Wall -fPIC -O2 -isystem /Users/certik/miniforge3/envs/pythran/include -arch arm64 -fPIC -O2 -isystem /Users/certik/miniforge3/envs/pythran/include -arch arm64 -DENABLE_PYTHON_MODULE -D__PYTHRAN__=3 -DPYTHRAN_BLAS_BLAS -I/Users/certik/miniforge3/envs/pythran/lib/python3.10/site-packages/pythran -I/Users/certik/miniforge3/envs/pythran/lib/python3.10/site-packages/numpy/core/include -I/Users/certik/miniforge3/envs/pythran/include -I/Users/certik/miniforge3/envs/pythran/include/python3.10 -c /var/folders/43/nspmtcpj5r589x58n_spl6jm000fsc/T/tmp0ikdp89m.cpp -o /var/folders/43/nspmtcpj5r589x58n_spl6jm000fsc/T/tmpra07avzl/var/folders/43/nspmtcpj5r589x58n_spl6jm000fsc/T/tmp0ikdp89m.o -std=c++11 -fno-math-errno -Wno-unused-function" failed with exit status 1

It doesn't show the actual error. But if I execute the command by hand, I get:

$ clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O2 -Wall -fPIC -O2 -isystem /Users/certik/miniforge3/envs/pythran/include -arch arm64 -fPIC -O2 -isystem /Users/certik/miniforge3/envs/pythran/include -arch arm64 -DENABLE_PYTHON_MODULE -D__PYTHRAN__=3 -DPYTHRAN_BLAS_BLAS -I/Users/certik/miniforge3/envs/pythran/lib/python3.10/site-packages/pythran -I/Users/certik/miniforge3/envs/pythran/lib/python3.10/site-packages/numpy/core/include -I/Users/certik/miniforge3/envs/pythran/include -I/Users/certik/miniforge3/envs/pythran/include/python3.10 -c /var/folders/43/nspmtcpj5r589x58n_spl6jm000fsc/T/tmp0ikdp89m.cpp -o /var/folders/43/nspmtcpj5r589x58n_spl6jm000fsc/T/tmpra07avzl/var/folders/43/nspmtcpj5r589x58n_spl6jm000fsc/T/tmp0ikdp89m.o -std=c++11 -fno-math-errno -Wno-unused-function
In file included from /var/folders/43/nspmtcpj5r589x58n_spl6jm000fsc/T/tmp0ikdp89m.cpp:1:
In file included from /Users/certik/miniforge3/envs/pythran/lib/python3.10/site-packages/pythran/pythonic/core.hpp:36:
In file included from /Users/certik/miniforge3/envs/pythran/lib/python3.10/site-packages/pythran/pythonic/types/slice.hpp:5:
In file included from /Users/certik/miniforge3/envs/pythran/lib/python3.10/site-packages/pythran/pythonic/types/NoneType.hpp:8:
In file included from /Users/certik/miniforge3/envs/pythran/lib/python3.10/site-packages/pythran/pythonic/builtins/bool_.hpp:7:
In file included from /Users/certik/miniforge3/envs/pythran/lib/python3.10/site-packages/pythran/pythonic/types/tuple.hpp:13:
In file included from /Users/certik/miniforge3/envs/pythran/lib/python3.10/site-packages/pythran/pythonic/types/ndarray.hpp:1303:
In file included from /Users/certik/miniforge3/envs/pythran/lib/python3.10/site-packages/pythran/pythonic/types/numpy_operators.hpp:15:
/Users/certik/miniforge3/envs/pythran/lib/python3.10/site-packages/pythran/pythonic/operator_/lshift.hpp:22:15: warning: converting the result of '<<' to a boolean; did you mean '(a << b) != 0'? [-Wint-in-bool-context]
      lshift, <<, (a <= (std::numeric_limits<decltype(b)>::max() >> b)))
              ^
1 warning generated.

I don't see an error, just a warning. So I don't know what the issue is.

serge-sans-paille commented 3 years ago

I don't see an error, just a warning. So I don't know what the issue is.

Can you try pythran -v dprod.py ?

certik commented 3 years ago

Sure, here is the output:

$ pythran -v dprod.py
INFO:      sys file exists: /Users/certik/miniforge3/envs/pythran/lib/python3.10/site-packages/pythran/pythran.cfg
INFO: platform file exists: /Users/certik/miniforge3/envs/pythran/lib/python3.10/site-packages/pythran/pythran-darwin.cfg
WARNING:     user file does not exist: /Users/certik/.pythranrc
INFO: pythranrc section [pythran] is valid and options are correct
INFO: pythranrc section [typing] is valid and options are correct
INFO: pythranrc section [compiler] is valid and options are correct
WARNING:root:No pythran specification, nothing will be exported
running build_ext
new_compiler returns <class 'distutils.unixccompiler.UnixCCompiler'>
building 'dprod' extension
C compiler: arm64-apple-darwin20.0.0-clang++ -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O2 -Wall -fPIC -O2 -isystem /Users/certik/miniforge3/envs/pythran/include -arch arm64 -fPIC -O2 -isystem /Users/certik/miniforge3/envs/pythran/include -arch arm64 -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -isystem /Users/certik/miniforge3/envs/pythran/include -D_FORTIFY_SOURCE=2 -isystem /Users/certik/miniforge3/envs/pythran/include

creating /var/folders/43/nspmtcpj5r589x58n_spl6jm000fsc/T/tmpjtcdjtic/var
creating /var/folders/43/nspmtcpj5r589x58n_spl6jm000fsc/T/tmpjtcdjtic/var/folders
creating /var/folders/43/nspmtcpj5r589x58n_spl6jm000fsc/T/tmpjtcdjtic/var/folders/43
creating /var/folders/43/nspmtcpj5r589x58n_spl6jm000fsc/T/tmpjtcdjtic/var/folders/43/nspmtcpj5r589x58n_spl6jm000fsc
creating /var/folders/43/nspmtcpj5r589x58n_spl6jm000fsc/T/tmpjtcdjtic/var/folders/43/nspmtcpj5r589x58n_spl6jm000fsc/T
compile options: '-DENABLE_PYTHON_MODULE -D__PYTHRAN__=3 -DPYTHRAN_BLAS_BLAS -I/Users/certik/miniforge3/envs/pythran/lib/python3.10/site-packages/pythran -I/Users/certik/miniforge3/envs/pythran/lib/python3.10/site-packages/numpy/core/include -I/Users/certik/miniforge3/envs/pythran/include -I/Users/certik/miniforge3/envs/pythran/include/python3.10 -c'
extra options: '-std=c++11 -fno-math-errno -Wno-unused-function -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -stdlib=libc++ -fvisibility-inlines-hidden -std=c++14 -fmessage-length=0 -isystem /Users/certik/miniforge3/envs/pythran/include'
arm64-apple-darwin20.0.0-clang++: /var/folders/43/nspmtcpj5r589x58n_spl6jm000fsc/T/tmpew0u0njj.cpp
In file included from /var/folders/43/nspmtcpj5r589x58n_spl6jm000fsc/T/tmpew0u0njj.cpp:1:
In file included from /Users/certik/miniforge3/envs/pythran/lib/python3.10/site-packages/pythran/pythonic/core.hpp:36:
In file included from /Users/certik/miniforge3/envs/pythran/lib/python3.10/site-packages/pythran/pythonic/types/slice.hpp:5:
In file included from /Users/certik/miniforge3/envs/pythran/lib/python3.10/site-packages/pythran/pythonic/types/NoneType.hpp:8:
In file included from /Users/certik/miniforge3/envs/pythran/lib/python3.10/site-packages/pythran/pythonic/builtins/bool_.hpp:7:
In file included from /Users/certik/miniforge3/envs/pythran/lib/python3.10/site-packages/pythran/pythonic/types/tuple.hpp:13:
In file included from /Users/certik/miniforge3/envs/pythran/lib/python3.10/site-packages/pythran/pythonic/types/ndarray.hpp:1303:
In file included from /Users/certik/miniforge3/envs/pythran/lib/python3.10/site-packages/pythran/pythonic/types/numpy_operators.hpp:15:
/Users/certik/miniforge3/envs/pythran/lib/python3.10/site-packages/pythran/pythonic/operator_/lshift.hpp:22:15: warning: converting the result of '<<' to a boolean; did you mean '(a << b) != 0'? [-Wint-in-bool-context]
      lshift, <<, (a <= (std::numeric_limits<decltype(b)>::max() >> b)))
              ^
1 warning generated.
arm64-apple-darwin20.0.0-clang++ -bundle -undefined dynamic_lookup -Wl,-rpath,/Users/certik/miniforge3/envs/pythran/lib -L/Users/certik/miniforge3/envs/pythran/lib -Wl,-rpath,/Users/certik/miniforge3/envs/pythran/lib -L/Users/certik/miniforge3/envs/pythran/lib -Wl,-pie -Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/Users/certik/miniforge3/envs/pythran/lib -L/Users/certik/miniforge3/envs/pythran/lib -ftree-vectorize -fPIC -fPIE -fstack-protector-strong -O2 -pipe -isystem /Users/certik/miniforge3/envs/pythran/include -D_FORTIFY_SOURCE=2 -isystem /Users/certik/miniforge3/envs/pythran/include /var/folders/43/nspmtcpj5r589x58n_spl6jm000fsc/T/tmpjtcdjtic/var/folders/43/nspmtcpj5r589x58n_spl6jm000fsc/T/tmpew0u0njj.o -L/Users/certik/miniforge3/envs/pythran/lib -lcblas -lblas -lcblas -lblas -o /var/folders/43/nspmtcpj5r589x58n_spl6jm000fsc/T/tmppi73t_lo/dprod.cpython-310-darwin.so -Wl,-pie -Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,/Users/certik/miniforge3/envs/pythran/lib -L/Users/certik/miniforge3/envs/pythran/lib
ld: warning: -pie being ignored. It is only used when linking a main executable
INFO: Generated module: dprod
INFO:pythran:Generated module: dprod
INFO: Output: /Users/certik/dprod.cpython-310-darwin.so
INFO:pythran:Output: /Users/certik/dprod.cpython-310-darwin.so
certik commented 3 years ago

So it seems to work? I assumed it didn't based on the error, but it seems it does work:

$ python
Python 3.10.0 | packaged by conda-forge | (default, Oct 12 2021, 21:22:54) [Clang 11.1.0 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import dprod
>>> dprod.__file__
'/Users/certik/dprod.cpython-310-darwin.so'
serge-sans-paille commented 3 years ago

This looks fine... at least for the second run. The 'critical' in the first run are less encouraging.

Please note this warning:

WARNING:root:No pythran specification, nothing will be exported

In order to do something useful with your module, you need a pythran export line (see https://pythran.readthedocs.io/en/latest/MANUAL.html#concerning-pythran-specifications)

certik commented 3 years ago

Ah, adding #pythran export dprod(int list, int list) fixed the problem, now it doesn't complain and it seems to work:

$ python
Python 3.10.0 | packaged by conda-forge | (default, Oct 12 2021, 21:22:54) [Clang 11.1.0 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import a
>>> a.dprod.__doc__
'Supported prototypes:\n\n    - dprod(int list, int list)'
>>> a.dprod([1,2], [3,4])
11

So I think this issue is fixed. But I think pythran should return more user friendly error that I forgot to add type annotation.

serge-sans-paille commented 3 years ago

Well, we already have a warning, I don't know what could be done differently?

certik commented 3 years ago

It seems if I forget to put in a type annotation, it should not fail in C++ as it does above? I thought it indicated there is some bug somewhere. If not, then this issue can be closed.

serge-sans-paille commented 3 years ago

1934 now properly raises an error in the scenario you're describing.Hopefully :-)

serge-sans-paille commented 3 years ago

Closing the issue as we now both have a decent error message and a solution to the original problem :-)