serge-sans-paille / pythran

Ahead of Time compiler for numeric kernels
https://pythran.readthedocs.io
BSD 3-Clause "New" or "Revised" License
1.99k stars 190 forks source link

-DUSE_XSIMD does not work with multidimensional arrays #1725

Open exenGT opened 3 years ago

exenGT commented 3 years ago

Here is a minimal example that generates this error:

pythran_test.py:

import numpy as np

# pythran export norming(float[][][])

def norming(np_array):
    norm_array = np.linalg.norm(np_array, axis=-1)
    return norm_array

Using the command pythran -vv pythran_test.py -DUSE_XSIMD, a long error message appears:

INFO:      sys file exists: /Users/jw598/miniconda3/lib/python3.7/site-packages/pythran/pythran.cfg
INFO: platform file exists: /Users/jw598/miniconda3/lib/python3.7/site-packages/pythran/pythran-darwin.cfg
INFO:     user file exists: /Users/jw598/.pythranrc
INFO: pythranrc section [pythran] is valid and options are correct
INFO: pythranrc section [typing] is valid and options are correct
INFO: pythranrc section [compiler] is valid and options are correct
running build_ext
new_compiler returns <class 'distutils.unixccompiler.UnixCCompiler'>
building 'pythran_test' extension
C compiler: g++ -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Users/jw598/miniconda3/include -arch x86_64 -I/Users/jw598/miniconda3/include -arch x86_64 -I/usr/local/opt/openblas/include

creating /tmp/tmpyujhd9es/tmp
compile options: '-DENABLE_PYTHON_MODULE -D__PYTHRAN__=3 -DUSE_XSIMD -DPYTHRAN_BLAS_OPENBLAS -I/usr/local/opt/openblas/include -I/Users/jw598/miniconda3/lib/python3.7/site-packages/pythran -I/Users/jw598/miniconda3/lib/python3.7/site-packages/numpy/core/include -I/Users/jw598/miniconda3/include/python3.7m -c'
extra options: '-std=c++11 -fno-math-errno -w'
g++: /tmp/tmph7adn9tl.cpp
In file included from /Users/jw598/miniconda3/lib/python3.7/site-packages/pythran/pythonic/types/ndarray.hpp:16,
                 from /Users/jw598/miniconda3/lib/python3.7/site-packages/pythran/pythonic/types/tuple.hpp:13,
                 from /Users/jw598/miniconda3/lib/python3.7/site-packages/pythran/pythonic/builtins/bool_.hpp:7,
                 from /Users/jw598/miniconda3/lib/python3.7/site-packages/pythran/pythonic/types/NoneType.hpp:8,
                 from /Users/jw598/miniconda3/lib/python3.7/site-packages/pythran/pythonic/types/slice.hpp:5,
                 from /Users/jw598/miniconda3/lib/python3.7/site-packages/pythran/pythonic/core.hpp:29,
                 from /tmp/tmph7adn9tl.cpp:1:
/Users/jw598/miniconda3/lib/python3.7/site-packages/pythran/pythonic/utils/broadcast_copy.hpp: In instantiation of 'void {anonymous}::pythonic::utils::vbroadcast_copy(E&&, const F&) [with vectorizer = {anonymous}::pythonic::types::vectorizer; E = {anonymous}::pythonic::types::numpy_iexpr<const {anonymous}::pythonic::types::ndarray<long int, {anonymous}::pythonic::types::array_base<long int, 2, {anonymous}::pythonic::types::tuple_version> >&>&; F = {anonymous}::pythonic::types::ndarray<long int, {anonymous}::pythonic::types::array_base<long int, 1, {anonymous}::pythonic::types::tuple_version> >]':
/Users/jw598/miniconda3/lib/python3.7/site-packages/pythran/pythonic/utils/broadcast_copy.hpp:247:48:   required from 'void {anonymous}::pythonic::utils::_broadcast_copy<{anonymous}::pythonic::types::vectorizer, 1, 0>::operator()(E&&, const F&) [with E = {anonymous}::pythonic::types::numpy_iexpr<const {anonymous}::pythonic::types::ndarray<long int, {anonymous}::pythonic::types::array_base<long int, 2, {anonymous}::pythonic::types::tuple_version> >&>&; F = {anonymous}::pythonic::types::ndarray<long int, {anonymous}::pythonic::types::array_base<long int, 1, {anonymous}::pythonic::types::tuple_version> >]'
/Users/jw598/miniconda3/lib/python3.7/site-packages/pythran/pythonic/utils/broadcast_copy.hpp:283:51:   required from 'void {anonymous}::pythonic::utils::broadcast_copy_dispatcher<E, F, N, D, true>::operator()(E&, const F&) [with E = {anonymous}::pythonic::types::numpy_iexpr<const {anonymous}::pythonic::types::ndarray<long int, {anonymous}::pythonic::types::array_base<long int, 2, {anonymous}::pythonic::types::tuple_version> >&>&; F = {anonymous}::pythonic::types::ndarray<long int, {anonymous}::pythonic::types::array_base<long int, 1, {anonymous}::pythonic::types::tuple_version> >; long unsigned int N = 1; long unsigned int D = 0]'
/Users/jw598/miniconda3/lib/python3.7/site-packages/pythran/pythonic/utils/broadcast_copy.hpp:291:59:   required from 'E& {anonymous}::pythonic::utils::broadcast_copy(E&, const F&) [with E = {anonymous}::pythonic::types::numpy_iexpr<const {anonymous}::pythonic::types::ndarray<long int, {anonymous}::pythonic::types::array_base<long int, 2, {anonymous}::pythonic::types::tuple_version> >&>&; F = {anonymous}::pythonic::types::ndarray<long int, {anonymous}::pythonic::types::array_base<long int, 1, {anonymous}::pythonic::types::tuple_version> >; long unsigned int N = 1; long unsigned int D = 0; bool vector_form = true]'
/Users/jw598/miniconda3/lib/python3.7/site-packages/pythran/pythonic/types/numpy_iexpr.hpp:64:51:   required from '{anonymous}::pythonic::types::numpy_iexpr<A>& {anonymous}::pythonic::types::numpy_iexpr<A>::operator=(const E&) [with E = {anonymous}::pythonic::types::ndarray<long int, {anonymous}::pythonic::types::array_base<long int, 1, {anonymous}::pythonic::types::tuple_version> >; Requires = void; Arg = const {anonymous}::pythonic::types::ndarray<long int, {anonymous}::pythonic::types::array_base<long int, 2, {anonymous}::pythonic::types::tuple_version> >&]'
/opt/local/include/gcc8/c++/bits/stl_algo.h:4304:12:   required from '_OIter std::transform(_IIter, _IIter, _OIter, _UnaryOperation) [with _IIter = {anonymous}::pythonic::types::numpy_expr_iterator<{anonymous}::pythonic::operator_::functor::ne, {anonymous}::pythonic::types::pshape<long int, long int>, {anonymous}::pythonic::types::const_nditerator<{anonymous}::pythonic::types::ndarray<double, {anonymous}::pythonic::types::pshape<long int, long int, long int> > >, {anonymous}::pythonic::types::const_broadcast_iterator<double> >; _OIter = {anonymous}::pythonic::types::nditerator<{anonymous}::pythonic::types::ndarray<long int, {anonymous}::pythonic::types::array_base<long int, 2, {anonymous}::pythonic::types::tuple_version> > >; _UnaryOperation = {anonymous}::pythonic::numpy::reduce(const E&, long int, {anonymous}::pythonic::types::none_type, Out&&) [with Op = {anonymous}::pythonic::operator_::functor::iadd; E = {anonymous}::pythonic::types::numpy_expr<{anonymous}::pythonic::operator_::functor::ne, {anonymous}::pythonic::types::ndarray<double, {anonymous}::pythonic::types::pshape<long int, long int, long int> >&, {anonymous}::pythonic::types::broadcast<double, double> >; Out = {anonymous}::pythonic::types::ndarray<long int, {anonymous}::pythonic::types::array_base<long int, 2, {anonymous}::pythonic::types::tuple_version> >&; typename std::enable_if<(E::value != 1), {anonymous}::pythonic::types::ndarray<typename std::conditional<(((std::is_integral<typename E::dtype>::value && (sizeof (typename E::dtype) < sizeof (long int))) && (! std::is_same<Op, {anonymous}::pythonic::operator_::functor::imin>::value)) && (! std::is_same<Op, {anonymous}::pythonic::operator_::functor::imax>::value)), typename std::conditional<std::is_same<typename E::dtype, bool>::value, long int, typename std::conditional<std::is_signed<typename E::dtype>::value, long int, long unsigned int>::type>::type, typename E::dtype>::type, {anonymous}::pythonic::types::array_base<long int, (E::value - 1), {anonymous}::pythonic::types::tuple_version> > >::type = {anonymous}::pythonic::types::ndarray<long int, {anonymous}::pythonic::types::array_base<long int, 2, {anonymous}::pythonic::types::tuple_version> >]::<lambda(std::iterator<std::random_access_iterator_tag, {anonymous}::pythonic::types::numpy_expr<{anonymous}::pythonic::operator_::functor::ne, {anonymous}::pythonic::types::numpy_iexpr<const {anonymous}::pythonic::types::ndarray<double, {anonymous}::pythonic::types::pshape<long int, long int, long int> >&>, {anonymous}::pythonic::types::broadcast<double, double> >, long int, {anonymous}::pythonic::types::numpy_expr<{anonymous}::pythonic::operator_::functor::ne, {anonymous}::pythonic::types::numpy_iexpr<const {anonymous}::pythonic::types::ndarray<double, {anonymous}::pythonic::types::pshape<long int, long int, long int> >&>, {anonymous}::pythonic::types::broadcast<double, double> >*, {anonymous}::pythonic::types::numpy_expr<{anonymous}::pythonic::operator_::functor::ne, {anonymous}::pythonic::types::numpy_iexpr<const {anonymous}::pythonic::types::ndarray<double, {anonymous}::pythonic::types::pshape<long int, long int, long int> >&>, {anonymous}::pythonic::types::broadcast<double, double> >&>::value_type)>]'
/Users/jw598/miniconda3/lib/python3.7/site-packages/pythran/pythonic/numpy/reduce.hpp:309:23:   [ skipping 3 instantiation contexts, use -ftemplate-backtrace-limit=0 to disable ]
/Users/jw598/miniconda3/lib/python3.7/site-packages/pythran/pythonic/include/numpy/sum.hpp:20:3:   required from 'decltype ({anonymous}::pythonic::numpy::sum((forward<Types>)(<unnamed>::pythonic::numpy::functor::sum::operator()::types)...)) {anonymous}::pythonic::numpy::functor::sum::operator()(Types&& ...) const [with Types = {{anonymous}::pythonic::types::numpy_expr<{anonymous}::pythonic::operator_::functor::ne, {anonymous}::pythonic::types::ndarray<double, {anonymous}::pythonic::types::pshape<long int, long int, long int> >&, {anonymous}::pythonic::types::broadcast<double, double> >, long int&}; decltype ({anonymous}::pythonic::numpy::sum((forward<Types>)(<unnamed>::pythonic::numpy::functor::sum::operator()::types)...)) = {anonymous}::pythonic::types::ndarray<long int, {anonymous}::pythonic::types::array_base<long int, 2, {anonymous}::pythonic::types::tuple_version> >]'
/Users/jw598/miniconda3/lib/python3.7/site-packages/pythran/pythonic/numpy/linalg/norm.hpp:59:47:   required from '{anonymous}::pythonic::numpy::linalg::norm_t<Array> {anonymous}::pythonic::numpy::linalg::norm(Array&&, double, long int) [with Array = {anonymous}::pythonic::types::ndarray<double, {anonymous}::pythonic::types::pshape<long int, long int, long int> >&; {anonymous}::pythonic::numpy::linalg::norm_t<Array> = {anonymous}::pythonic::types::ndarray<double, {anonymous}::pythonic::types::array_base<long int, 2, {anonymous}::pythonic::types::tuple_version> >]'
/Users/jw598/miniconda3/lib/python3.7/site-packages/pythran/pythonic/numpy/linalg/norm.hpp:80:18:   required from '{anonymous}::pythonic::numpy::linalg::norm_t<Array> {anonymous}::pythonic::numpy::linalg::norm(Array&&, {anonymous}::pythonic::types::none_type, double) [with Array = {anonymous}::pythonic::types::ndarray<double, {anonymous}::pythonic::types::pshape<long int, long int, long int> >&; {anonymous}::pythonic::numpy::linalg::norm_t<Array> = {anonymous}::pythonic::types::ndarray<double, {anonymous}::pythonic::types::array_base<long int, 2, {anonymous}::pythonic::types::tuple_version> >]'
/Users/jw598/miniconda3/lib/python3.7/site-packages/pythran/pythonic/include/numpy/linalg/norm.hpp:49:5:   required from 'decltype ({anonymous}::pythonic::numpy::linalg::norm((forward<Types>)(<unnamed>::pythonic::numpy::linalg::functor::norm::operator()::types)...)) {anonymous}::pythonic::numpy::linalg::functor::norm::operator()(Types&& ...) const [with Types = {{anonymous}::pythonic::types::ndarray<double, {anonymous}::pythonic::types::pshape<long int, long int, long int> >&, const {anonymous}::pythonic::types::none_type&, long int}; decltype ({anonymous}::pythonic::numpy::linalg::norm((forward<Types>)(<unnamed>::pythonic::numpy::linalg::functor::norm::operator()::types)...)) = {anonymous}::pythonic::types::ndarray<double, {anonymous}::pythonic::types::array_base<long int, 2, {anonymous}::pythonic::types::tuple_version> >]'
/tmp/tmph7adn9tl.cpp:39:196:   required from 'typename __pythran_pythran_test::norming::type<argument_type0>::result_type __pythran_pythran_test::norming::operator()(argument_type0&&) const [with argument_type0 = {anonymous}::pythonic::types::ndarray<double, {anonymous}::pythonic::types::pshape<long int, long int, long int> >&; typename __pythran_pythran_test::norming::type<argument_type0>::result_type = {anonymous}::pythonic::types::ndarray<double, {anonymous}::pythonic::types::array_base<long int, 2, {anonymous}::pythonic::types::tuple_version> >]'
/tmp/tmph7adn9tl.cpp:50:86:   required from here
/Users/jw598/miniconda3/lib/python3.7/site-packages/pythran/pythonic/utils/broadcast_copy.hpp:202:30: error: 'size' is not a member of 'vT' {aka 'long int'}
     static const std::size_t vN = vT::size;
                              ^~
/Users/jw598/miniconda3/lib/python3.7/site-packages/pythran/pythonic/utils/broadcast_copy.hpp:221:20: error: invalid use of incomplete type 'xsimd::simd_return_type<long int, long int, 1>' {aka 'class xsimd::batch<long int, 1>'}
         iter.store(*oiter);
                    ^~~~~~
In file included from /Users/jw598/miniconda3/lib/python3.7/site-packages/pythran/xsimd/types/../types/xsimd_base.hpp:25,
                 from /Users/jw598/miniconda3/lib/python3.7/site-packages/pythran/xsimd/types/xsimd_traits.hpp:16,
                 from /Users/jw598/miniconda3/lib/python3.7/site-packages/pythran/xsimd/xsimd.hpp:16,
                 from /Users/jw598/miniconda3/lib/python3.7/site-packages/pythran/pythonic/include/types/nditerator.hpp:7,
                 from /Users/jw598/miniconda3/lib/python3.7/site-packages/pythran/pythonic/include/types/tuple.hpp:6,
                 from /Users/jw598/miniconda3/lib/python3.7/site-packages/pythran/pythonic/include/builtins/bool_.hpp:5,
                 from /Users/jw598/miniconda3/lib/python3.7/site-packages/pythran/pythonic/builtins/bool_.hpp:4,
                 from /Users/jw598/miniconda3/lib/python3.7/site-packages/pythran/pythonic/types/NoneType.hpp:8,
                 from /Users/jw598/miniconda3/lib/python3.7/site-packages/pythran/pythonic/types/slice.hpp:5,
                 from /Users/jw598/miniconda3/lib/python3.7/site-packages/pythran/pythonic/core.hpp:29,
                 from /tmp/tmph7adn9tl.cpp:1:
/Users/jw598/miniconda3/lib/python3.7/site-packages/pythran/xsimd/types/../types/xsimd_utils.hpp:26:11: note: declaration of 'xsimd::simd_return_type<long int, long int, 1>' {aka 'class xsimd::batch<long int, 1>'}
     class batch;
           ^~~~~
WARNING: Compilation error, trying hard to find its origin...
WARNING: Nop, I'm going to flood you with C++ errors!
CRITICAL: Cover me Jack. Jack? Jaaaaack!!!!
E: error: Command "g++ -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/Users/jw598/miniconda3/include -arch x86_64 -I/Users/jw598/miniconda3/include -arch x86_64 -I/usr/local/opt/openblas/include -DENABLE_PYTHON_MODULE -D__PYTHRAN__=3 -DUSE_XSIMD -DPYTHRAN_BLAS_OPENBLAS -I/usr/local/opt/openblas/include -I/Users/jw598/miniconda3/lib/python3.7/site-packages/pythran -I/Users/jw598/miniconda3/lib/python3.7/site-packages/numpy/core/include -I/Users/jw598/miniconda3/include/python3.7m -c /tmp/tmph7adn9tl.cpp -o /tmp/tmpyujhd9es/tmp/tmph7adn9tl.o -std=c++11 -fno-math-errno -w" failed with exit status 1

My system configuration is:

OS: MacOS Catalina 10.15.7 gcc: 10.2.0 python: 3.7.7 pythran: 0.9.8.post2 transonic: 0.4.7.post0

Thanks a lot!

Jingyang Wang

serge-sans-paille commented 3 years ago

Thanks for the detailed report. Your example compiles fine on my side, but I'm running a Linux host, so I'd say it's an architecture-dependent issue. @JohanMabille does that ring any bell?

JohanMabille commented 3 years ago

Yes, the hell of long ;)

@serge-sans-paille Can you define XSIMD_ENABLE_FALLBACK in Pythran (or have an option that let the user define it)? This would allow to use the fallback batch (that is, batch<X, 1>) when the library does not find a specialiazed batch for the considered type. I guess in this particular case, this would be a workaround since vectorization should work with long int.

@exenGT can you compile and run the following code:

#include <cstdint>
#include <iostream>
#include <type_traits>

int main(int argc, char* argv)
{
    std::cout << "size of long: " << sizeof(long) << std::endl;
    std::cout << "long == int: " << std::is_same<long, int>::value << std::endl;
    std::cout << "lont == long long: " << std::is_same<long, long long>::value << std::endl;
    std::cout << "long == int32_t: " << std::is_same<long, int32_t>::value << std::endl;
    std::cout << "long == int64_t: " << std::is_same<long, int64_t>::value << std::endl;
    return 0;
}

@serge-sans-paille I remember asking the same to another user here, but I could not retrieve the issue (just in case of you remember it too)

exenGT commented 3 years ago

Hi @JohanMabille, the output of your code on my machine is:

size of long: 8
long == int: 0
lont == long long: 0
long == int32_t: 0
long == int64_t: 0

Hope that helps!

JohanMabille commented 3 years ago

@exenGT thanks, that's definitely helpful! (and super weird Oo)

edmBernard commented 3 years ago

I'm not really sure it's exactly the same issue, I got a simillar error with this example using USE_XSIMD

# pythran export mysum(int[])  # failed
# pythran export mysum(int8[])  # work
# pythran export mysum(int16[])  # work
# pythran export mysum(int32[])  # failed
# pythran export mysum(int64[])  # work
# pythran export mysum(float[])  # failed and all variant of float also fail
def mysum(v0):
    return v0 + 1

OS: Windows 10 clang-cl python: 3.7.6 and 3.9 pythran: 0.9.8.post2 and master

cycomanic commented 3 years ago

@JohanMabille sorry quite a bit delay. You asked me for that info in #1646 (my answer is provided there).