search

bpartridge / PyPhi

Python for Scientific Computing on Intel's Xeon Phi MIC Platform
22 stars 7 forks source link

Numpy compiling error #4

Open rkawsar opened 9 years ago

rkawsar commented 9 years ago

Hello, I have limited knowledge about Xeon Phi system and I am trying to compile python, numpy and scipy using PyPhi. My system is a Ubuntu 14.04 (64x) with composer_xe_2015 and host system python is 2.7.6 (and have adopted "site.cfg" for 'composer_xe_2015'). but I encountered an error while running: "cd numpy && PYTHONXCPREFIX=$(MIC_PY_HOME) python numpyxc.py build_clib" ... and here's what the error looks like. any idea, what I am doing wrong?

part of my log

make[1]: Leaving directory `/opt/pyphi/mic/python' wget http://downloads.sourceforge.net/project/numpy/NumPy/1.8.1/numpy-1.8.1.tar.gz -O numpy.tgz --2015-10-09 15:52:05-- http://downloads.sourceforge.net/project/numpy/NumPy/1.8.1/numpy-1.8.1.tar.gz Resolving downloads.sourceforge.net (downloads.sourceforge.net)... 216.34.181.59 Connecting to downloads.sourceforge.net (downloads.sourceforge.net)|216.34.181.59|:80... connected. HTTP request sent, awaiting response... 302 Found Location: http://netassist.dl.sourceforge.net/project/numpy/NumPy/1.8.1/numpy-1.8.1.tar.gz [following] --2015-10-09 15:52:06-- http://netassist.dl.sourceforge.net/project/numpy/NumPy/1.8.1/numpy-1.8.1.tar.gz Resolving netassist.dl.sourceforge.net (netassist.dl.sourceforge.net)... 62.205.134.42, 2a01:d0:0:37::2 Connecting to netassist.dl.sourceforge.net (netassist.dl.sourceforge.net)|62.205.134.42|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 3794076 (3.6M) [application/x-gzip] Saving to: ‘numpy.tgz’

100%[================================================================================================================================================>] 3,794,076 --.-K/s in 0.03s

2015-10-09 15:52:18 (104 MB/s) - ‘numpy.tgz’ saved [3794076/3794076]

mkdir numpy tar -xf numpy.tgz -C numpy --strip-components=1 wget https://raw.github.com/bpartridge/PyPhi/master/numpyxc.py -O numpy/numpyxc.py --2015-10-09 15:52:18-- https://raw.github.com/bpartridge/PyPhi/master/numpyxc.py Resolving raw.github.com (raw.github.com)... 185.31.17.133 Connecting to raw.github.com (raw.github.com)|185.31.17.133|:443... connected. HTTP request sent, awaiting response... 301 Moved Permanently Location: https://raw.githubusercontent.com/bpartridge/PyPhi/master/numpyxc.py [following] --2015-10-09 15:52:19-- https://raw.githubusercontent.com/bpartridge/PyPhi/master/numpyxc.py Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.31.17.133 Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.31.17.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 2602 (2.5K) [text/plain] Saving to: ‘numpy/numpyxc.py’

100%[================================================================================================================================================>] 2,602 --.-K/s in 0s

2015-10-09 15:52:19 (482 MB/s) - ‘numpy/numpyxc.py’ saved [2602/2602]

wget https://raw.githubusercontent.com/neel9102/michack/master/PyPhi/site.cfg -O numpy/site.cfg --2015-10-09 15:52:19-- https://raw.githubusercontent.com/neel9102/michack/master/PyPhi/site.cfg Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.31.17.133 Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.31.17.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 223 [text/plain] Saving to: ‘numpy/site.cfg’

100%[================================================================================================================================================>] 223 --.-K/s in 0s

2015-10-09 15:52:19 (45.1 MB/s) - ‘numpy/site.cfg’ saved [223/223]

wget https://raw.github.com/bpartridge/PyPhi/master/setup.cfg -O numpy/setup.cfg --2015-10-09 15:52:19-- https://raw.github.com/bpartridge/PyPhi/master/setup.cfg Resolving raw.github.com (raw.github.com)... 185.31.17.133 Connecting to raw.github.com (raw.github.com)|185.31.17.133|:443... connected. HTTP request sent, awaiting response... 301 Moved Permanently Location: https://raw.githubusercontent.com/bpartridge/PyPhi/master/setup.cfg [following] --2015-10-09 15:52:19-- https://raw.githubusercontent.com/bpartridge/PyPhi/master/setup.cfg Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.31.17.133 Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.31.17.133|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 193 [text/plain] Saving to: ‘numpy/setup.cfg’

100%[================================================================================================================================================>] 193 --.-K/s in 0s

2015-10-09 15:52:20 (38.2 MB/s) - ‘numpy/setup.cfg’ saved [193/193]

build_clib forces config.h to be created; it needs to be modified before extensions are built

cd numpy && PYTHONXCPREFIX=/opt/pyphi/mic/python/_install python numpyxc.py build_clib /opt/pyphi/mic/numpy/setup.py Running from numpy source directory. /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'test_suite' warnings.warn(msg) /bin/sh: 1: svnversion: not found F2PY Version 2 blas_opt_info: blas_mkl_info: FOUND: libraries = ['mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'pthread'] library_dirs = ['/opt/intel/composer_xe_2015/mkl/lib/mic'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/intel/composer_xe_2015/mkl/include']

FOUND: libraries = ['mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'pthread'] library_dirs = ['/opt/intel/composer_xe_2015/mkl/lib/mic'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/intel/composer_xe_2015/mkl/include']

/bin/sh: 1: svnversion: not found non-existing path in 'numpy/lib': 'benchmarks' lapack_opt_info: openblas_info: libraries not found in ['/usr/local/lib', '/usr/lib64', '/usr/lib', '/usr/lib/x86_64-linux-gnu'] NOT AVAILABLE

lapack_mkl_info: mkl_info: FOUND: libraries = ['mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'pthread'] library_dirs = ['/opt/intel/composer_xe_2015/mkl/lib/mic'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/intel/composer_xe_2015/mkl/include']

FOUND: libraries = ['mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'pthread'] library_dirs = ['/opt/intel/composer_xe_2015/mkl/lib/mic'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/intel/composer_xe_2015/mkl/include']

FOUND: libraries = ['mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'pthread'] library_dirs = ['/opt/intel/composer_xe_2015/mkl/lib/mic'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/intel/composer_xe_2015/mkl/include']

/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'define_macros' warnings.warn(msg) running build_clib running build_src build_src building py_modules sources creating build creating build/src.linux-x86_64-2.7 creating build/src.linux-x86_64-2.7/numpy creating build/src.linux-x86_64-2.7/numpy/distutils building library "npymath" sources customize IntelMICFCompiler Found executable /opt/intel/bin/ifort customize IntelMICFCompiler using config C compiler: icc -mmic -mkl -fPIC -fp-model strict -g -I/opt/pyphi/mic/python/_install/include -L/opt/pyphi/mic/python/_install/lib -L/opt/pyphi/mic/python/_install/lib/python2.7 -L/opt/pyphi/mic/python/_install/lib/python2.7/site-packages/numpy/core/lib

compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/usr/include/python2.7 -c' icc: _configtest.c icc: command line remark #10364: Using '-fp-model strict' forces the compiler to generate legacy floating-point instructions instead of KNC for most floating-point operations, which may impact performance. Consider using a floating-point model that allows the compiler to assume the default floating-point environment. icc -mmic -mkl -fPIC -fp-model strict -g -I/opt/pyphi/mic/python/_install/include -L/opt/pyphi/mic/python/_install/lib -L/opt/pyphi/mic/python/_install/lib/python2.7 -L/opt/pyphi/mic/python/_install/lib/python2.7/site-packages/numpy/core/lib _configtest.o -o _configtest icc: command line remark #10364: Using '-fp-model strict' forces the compiler to generate legacy floating-point instructions instead of KNC for most floating-point operations, which may impact performance. Consider using a floating-point model that allows the compiler to assume the default floating-point environment. success! removing: _configtest.c _configtest.o _configtest C compiler: icc -mmic -mkl -fPIC -fp-model strict -g -I/opt/pyphi/mic/python/_install/include -L/opt/pyphi/mic/python/_install/lib -L/opt/pyphi/mic/python/_install/lib/python2.7 -L/opt/pyphi/mic/python/_install/lib/python2.7/site-packages/numpy/core/lib

compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/usr/include/python2.7 -c' icc: _configtest.c icc: command line remark #10364: Using '-fp-model strict' forces the compiler to generate legacy floating-point instructions instead of KNC for most floating-point operations, which may impact performance. Consider using a floating-point model that allows the compiler to assume the default floating-point environment. icc -mmic -mkl -fPIC -fp-model strict -g -I/opt/pyphi/mic/python/_install/include -L/opt/pyphi/mic/python/_install/lib -L/opt/pyphi/mic/python/_install/lib/python2.7 -L/opt/pyphi/mic/python/_install/lib/python2.7/site-packages/numpy/core/lib _configtest.o -o _configtest icc: command line remark #10364: Using '-fp-model strict' forces the compiler to generate legacy floating-point instructions instead of KNC for most floating-point operations, which may impact performance. Consider using a floating-point model that allows the compiler to assume the default floating-point environment. success! removing: _configtest.c _configtest.o _configtest creating build/src.linux-x86_64-2.7/numpy/core creating build/src.linux-x86_64-2.7/numpy/core/src creating build/src.linux-x86_64-2.7/numpy/core/src/npymath conv_template:> build/src.linux-x86_64-2.7/numpy/core/src/npymath/npy_math.c conv_template:> build/src.linux-x86_64-2.7/numpy/core/src/npymath/ieee754.c conv_template:> build/src.linux-x86_64-2.7/numpy/core/src/npymath/npy_math_complex.c building library "npysort" sources creating build/src.linux-x86_64-2.7/numpy/core/src/npysort conv_template:> build/src.linux-x86_64-2.7/numpy/core/src/npysort/quicksort.c conv_template:> build/src.linux-x86_64-2.7/numpy/core/src/npysort/mergesort.c conv_template:> build/src.linux-x86_64-2.7/numpy/core/src/npysort/heapsort.c creating build/src.linux-x86_64-2.7/numpy/core/src/private conv_template:> build/src.linux-x86_64-2.7/numpy/core/src/private/npy_partition.h adding 'build/src.linux-x86_64-2.7/numpy/core/src/private' to include_dirs. conv_template:> build/src.linux-x86_64-2.7/numpy/core/src/npysort/selection.c None - nothing done with h_files = ['build/src.linux-x86_64-2.7/numpy/core/src/private/npy_partition.h'] building extension "numpy.core._dummy" sources Generating build/src.linux-x86_64-2.7/numpy/core/include/numpy/config.h C compiler: icc -mmic -mkl -fPIC -fp-model strict -g -I/opt/pyphi/mic/python/_install/include -L/opt/pyphi/mic/python/_install/lib -L/opt/pyphi/mic/python/_install/lib/python2.7 -L/opt/pyphi/mic/python/_install/lib/python2.7/site-packages/numpy/core/lib

compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -Inumpy/core/include -I/usr/include/python2.7 -c' icc: _configtest.c icc: command line remark #10364: Using '-fp-model strict' forces the compiler to generate legacy floating-point instructions instead of KNC for most floating-point operations, which may impact performance. Consider using a floating-point model that allows the compiler to assume the default floating-point environment. In file included from /usr/include/python2.7/Python.h(8), from _configtest.c(1): /usr/include/python2.7/pyconfig.h(3): catastrophic error: cannot open source file "x86_64-linux-gnu/python2.7/pyconfig.h"

include <x86_64-linux-gnu/python2.7/pyconfig.h>

                                                ^

compilation aborted for _configtest.c (code 4) icc: command line remark #10364: Using '-fp-model strict' forces the compiler to generate legacy floating-point instructions instead of KNC for most floating-point operations, which may impact performance. Consider using a floating-point model that allows the compiler to assume the default floating-point environment. In file included from /usr/include/python2.7/Python.h(8), from _configtest.c(1): /usr/include/python2.7/pyconfig.h(3): catastrophic error: cannot open source file "x86_64-linux-gnu/python2.7/pyconfig.h"

include <x86_64-linux-gnu/python2.7/pyconfig.h>

                                                ^

compilation aborted for _configtest.c (code 4) failure. removing: _configtest.c _configtest.o Traceback (most recent call last): File "numpyxc.py", line 73, in setup.setup_package() File "/opt/pyphi/mic/numpy/setup.py", line 231, in setup_package setup(metadata) File "/opt/pyphi/mic/numpy/numpy/distutils/core.py", line 169, in setup return old_setup(new_attr) File "/usr/lib/python2.7/distutils/core.py", line 151, in setup dist.run_commands() File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/opt/pyphi/mic/numpy/numpy/distutils/command/build_clib.py", line 63, in run self.run_command('build_src') File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command self.distribution.run_command(command) File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/opt/pyphi/mic/numpy/numpy/distutils/command/build_src.py", line 153, in run self.build_sources() File "/opt/pyphi/mic/numpy/numpy/distutils/command/build_src.py", line 170, in build_sources self.build_extension_sources(ext) File "/opt/pyphi/mic/numpy/numpy/distutils/command/build_src.py", line 329, in build_extension_sources sources = self.generate_sources(sources, ext) File "/opt/pyphi/mic/numpy/numpy/distutils/command/build_src.py", line 386, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 428, in generate_config_h moredefs, ignored = cocache.check_types(config_cmd, ext, build_dir) File "numpy/core/setup.py", line 42, in check_types out = check_types(_a, *_kw) File "numpy/core/setup.py", line 289, in check_types "Cannot compile 'Python.h'. Perhaps you need to "\ SystemError: Cannot compile 'Python.h'. Perhaps you need to install python-dev|python-devel. make: *\ [numpyxc] Error 1

bpartridge commented 9 years ago

I no longer have access to Xeon Phi infrastructure, so I haven't been maintaining this. Looks like it's trying to use the host system's Python header files rather than the MIC-ready ones in /opt/pyphi/mic/python, even though the C compiler seems to be set correctly with -I/opt/pyphi/mic/python/_install/include. Make sure that there are indeed Python header files at that path? Other than that, I can't replicate the bug on my end, so I'm afraid there's little else I can do.