search

firedrakeproject / firedrake

Firedrake is an automated system for the portable solution of partial differential equations using the finite element method (FEM)
https://firedrakeproject.org
Other
505 stars 159 forks source link

firedrake-install tries building PyOP2 including MPI with icc #1280

Closed agzimmerman closed 5 years ago

agzimmerman commented 6 years ago

Please let me know if there's a more appropriate place to report installation issues. I am trying to install Firedrake for the first time.

I'm installing as a user on CentOS 7.4.

I run the installation script as follows

curl -O https://raw.githubusercontent.com/firedrakeproject/firedrake/master/scripts/firedrake-install

env -u PYTHONPATH python3 firedrake-install --no-package-manager

I added "--no-package-manager" because digging through closed issues on here led me to how this is needed for anything other than Ubuntu. I added "env -u PYTHONPATH" because otherwise the installer throws an error about how I probably shouldn't be setting my own PYTHONPATH.

The installation fails with the following:

2018-09-04 16:37:55,538 INFO Installing PyOP2/ 2018-09-04 16:37:55,539 DEBUG Running command '/rwthfs/rz/cluster/home/az916605/firedrake/firedr ake/bin/python -m pip install --no-binary mpi4py --no-deps -vvv -e PyOP2/' 2018-09-04 16:38:12,756 DEBUG Created temporary directory: /w0/tmp/az916605/cluster_3145/pip-eph em-wheel-cache-x8t19nyq Created temporary directory: /w0/tmp/az916605/cluster_3145/pip-req-tracker-e3j93z9n Created requirements tracker '/w0/tmp/az916605/cluster_3145/pip-req-tracker-e3j93z9n' Created temporary directory: /w0/tmp/az916605/cluster_3145/pip-install-hd8116wt Obtaining file:///rwthfs/rz/cluster/home/az916605/firedrake/firedrake/src/PyOP2 Added file:///rwthfs/rz/cluster/home/az916605/firedrake/firedrake/src/PyOP2 to build tracker '/ w0/tmp/az916605/cluster_3145/pip-req-tracker-e3j93z9n' Running setup.py (path:/rwthfs/rz/cluster/home/az916605/firedrake/firedrake/src/PyOP2/setup.py) egg_info for package from file:///rwthfs/rz/cluster/home/az916605/firedrake/firedrake/src/PyOP2 Running command python setup.py egg_info running egg_info creating PyOP2.egg-info writing PyOP2.egg-info/PKG-INFO writing dependency_links to PyOP2.egg-info/dependency_links.txt writing requirements to PyOP2.egg-info/requires.txt writing top-level names to PyOP2.egg-info/top_level.txt writing manifest file 'PyOP2.egg-info/SOURCES.txt' reading manifest file 'PyOP2.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' /usr/local_rwth/sw/python/3.6.0/x86_64/lib/python3.6/distutils/dist.py:261: UserWarning: Unkn own distribution option: 'test_requires' warnings.warn(msg) warning: no files found matching '.c' under directory 'pyop2' writing manifest file 'PyOP2.egg-info/SOURCES.txt' Source in ./PyOP2 has version 0.13.0+394.ge831b59, which satisfies requirement PyOP2==0.13.0+39 4.ge831b59 from file:///rwthfs/rz/cluster/home/az916605/firedrake/firedrake/src/PyOP2 Removed PyOP2==0.13.0+394.ge831b59 from file:///rwthfs/rz/cluster/home/az916605/firedrake/fired rake/src/PyOP2 from build tracker '/w0/tmp/az916605/cluster_3145/pip-req-tracker-e3j93z9n' Installing collected packages: PyOP2 Running setup.py develop for PyOP2 Running command /rwthfs/rz/cluster/home/az916605/firedrake/firedrake/bin/python -c "import se tuptools, tokenize;file='/rwthfs/rz/cluster/home/az916605/firedrake/firedrake/src/PyOP2/setup .py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();ex ec(compile(code, file, 'exec'))" develop --no-deps running develop running egg_info writing PyOP2.egg-info/PKG-INFO writing dependency_links to PyOP2.egg-info/dependency_links.txt writing requirements to PyOP2.egg-info/requires.txt writing top-level names to PyOP2.egg-info/top_level.txt reading manifest file 'PyOP2.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' /usr/local_rwth/sw/python/3.6.0/x86_64/lib/python3.6/distutils/dist.py:261: UserWarning: Unknown distribution option: 'test_requires' warnings.warn(msg) warning: no files found matching '.c' under directory 'pyop2' writing manifest file 'PyOP2.egg-info/SOURCES.txt' running build_ext cythoning pyop2/sparsity.pyx to pyop2/sparsity.c building 'pyop2.sparsity' extension creating build creating build/temp.linux-x86_64-3.6 creating build/temp.linux-x86_64-3.6/pyop2 icc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Ipyop2 -I/rwthfs/rz/cluster/home/az916605/firedrake/firedrake/lib/python3.6/site-packages/numpy/core/include -I/rwthfs/rz/cluster/home/az916605/firedrake/firedrake/lib/python3.6/site-packages/petsc4py/include -I/rwthfs/rz/cluster/home/az916605/firedrake/firedrake/lib/python3.6/site-packages/petsc/include -I/rwthfs/rz/cluster/home/az916605/firedrake/firedrake/include -I/usr/local_rwth/sw/python/3.6.0/x86_64/include/python3.6m -c pyop2/sparsity.c -o build/temp.linux-x86_64-3.6/pyop2/sparsity.o icc: command line warning #10006: ignoring unknown option '-Wno-unused-result' In file included from /rwthfs/rz/cluster/home/az916605/firedrake/firedrake/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h(1821), from /rwthfs/rz/cluster/home/az916605/firedrake/firedrake/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h(18), from /rwthfs/rz/cluster/home/az916605/firedrake/firedrake/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h(4), from pyop2/sparsity.c(567): /rwthfs/rz/cluster/home/az916605/firedrake/firedrake/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h(15): warning #1224: #warning directive: "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION"

warning "Using deprecated NumPy API, disable it by " \

   ^

In file included from /rwthfs/rz/cluster/home/az916605/firedrake/firedrake/lib/python3.6/site-packages/petsc/include/petscbag.h(4),
                 from /rwthfs/rz/cluster/home/az916605/firedrake/firedrake/lib/python3.6/site-packages/petsc/include/petsc.h(5),
                 from pyop2/sparsity.c(569):
/rwthfs/rz/cluster/home/az916605/firedrake/firedrake/lib/python3.6/site-packages/petsc/include/petscsys.h(132): catastrophic error: cannot open source file "mpi.h"
  #  include <mpi.h>
                    ^
compilation aborted for pyop2/sparsity.c (code 4)
error: command 'icc' failed with exit status 4

So to me it appears that icc was used where perhaps mpicc was supposed to be used instead. Maybe I'm entirely wrong about this and we should edit the name of this issue.

Any advice on how I should proceed? I'm not sure how much I should dig into this on my own.

Also for reference:

which mpicc

/opt/MPI/openmpi-1.10.4/linux/intel_16.0.2.181/bin/mpicc

which icc

/opt/intel/Compiler/16.0/2.181/rwthlnk/bin/intel64/icc

miklos1 commented 6 years ago

So to me it appears that icc was used where perhaps mpicc was supposed to be used instead. Maybe I'm entirely wrong about this and we should edit the name of this issue.

That sounds like it. Why does it use icc though? What are your environment variables? You should use mpicc though, and if you want to use a different compiler, then set it for your MPI implementation.

For example, if you use OpenMPI, then you set OMPI_CC for your favourite compiler, if you use MPICH, then you set MPICH_CC to your favourite compiler. If you use Intel MPI, then look up into your manual.

agzimmerman commented 6 years ago

Why does it use icc though?

@miklos1 , that's exactly my question. I didn't change any of the MPI environment. Looking through firedrake-install.log, I see MPICC=mpicc and MPICXX=mpicxx. So to me this looks like a bug, though I assume it's more likely that I'm doing something wrong.

miklos1 commented 6 years ago

 What are your environment variables?I meant you could report the output of printenv...On 12 Sep 2018, at 11:56, Alexander G. Zimmerman notifications@github.com wrote:Why does it use icc though? @miklos1 , that's exactly my question. I didn't change any of the MPI environment. Looking through firedrake-install.log, I see MPICC=mpicc and MPICXX=mpicxx. So to me this looks like a bug, though I assume it's more likely that I'm doing something wrong.—You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub, or mute the thread.

miklos1 commented 6 years ago

Ugh, GitHub formatted the previous email reply very wrong.

Please report the output of printenv!

On 12 Sep 2018, at 11:56, Alexander G. Zimmerman notifications@github.com wrote:

Why does it use icc though?

@miklos1 https://github.com/miklos1 , that's exactly my question. I didn't change any of the MPI environment. Looking through firedrake-install.log, I see MPICC=mpicc and MPICXX=mpicxx. So to me this looks like a bug, though I assume it's more likely that I'm doing something wrong.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/firedrakeproject/firedrake/issues/1280#issuecomment-420586650, or mute the thread https://github.com/notifications/unsubscribe-auth/AEBAmcwmkQUzhOKszLDUk5BJRZb0K0yvks5uaNovgaJpZM4WZMlv.

agzimmerman commented 6 years ago

Here's the output of printenv:

LANG=C.UTF-8 USER=az916605 LOGNAME=az916605 HOME=/home/az916605 PATH=/usr/local_rwth/sw/python/3.6.0/x86_64/bin:/usr/local_rwth/sw/python/3.6.0/x86_64/extra/bin:/home/az916605/apps//bin:/opt/MPI/bin:/opt/MPI/openmpi-1.10.4/linux/intel_16.0.2.181/bin:/opt/intel/Compiler/16.0/2.181/debugger_2016/gdb/intel64/bin:/opt/intel/Compiler/16.0/2.181/debugger_2016/gdb/intel64_mic/bin:/opt/intel/Compiler/16.0/2.181/rwthlnk/bin/intel64:/opt/lsf/10.1/linux3.10-glibc2.17-x86_64/etc:/opt/lsf/10.1/linux3.10-glibc2.17-x86_64/bin:/usr/local_host/bin:/usr/local_host/sbin:/usr/local_rwth/bin:/usr/local_rwth/sbin:/usr/bin:/usr/sbin:/bin:/sbin:/usr/local/bin:/usr/local/sbin:. MAIL=/var/mail/az916605 SHELL=/usr/local_rwth/bin/zsh TZ=:/etc/localtime SSH_CLIENT=134.61.189.152 63032 22 SSH_CONNECTION=134.61.189.152 63032 134.61.193.128 22 SSH_TTY=/dev/pts/190 TERM=xterm-256color KRB5CCNAME=FILE:/tmp/krb5cc_27443_LO7ma3VBnA SHLVL=1 PWD=/home/az916605/firedrake OLDPWD=/home/az916605 R_SHLVL=1 R_LOGNAME=az916605 R_LOGINNAME=az916605 WORK=/work/az916605 HPCWORK=/hpcwork/az916605 HOME_SNAPSHOT=/home/az916605/.snapshot WORK_SNAPSHOT=/work/az916605/.snapshot/.snapshot HOSTNAME=cluster.rz.RWTH-Aachen.DE R_LOGINTAG=37641@cluster.rz.RWTH-Aachen.DE R_LOGINPID=37641 LOGINPID=37641 R_LOGINMODE=LOGIN TMP=/tmp/az916605/cluster_37641 TEMP=/tmp/az916605/cluster_37641 TMPDIR=/tmp/az916605/cluster_37641 TMPSESS=/tmp/az916605/cluster_37641 R_CPUGROUPS=beckton,nehalem R_GROUPS=bench_ls,all_nets,all_users,autoreboot,print,ssh_gssapi_authen,cgmv,perfmon,dialog,lustreclient,rpmmaster,modules40,lsf,lsf_sbm,lsf_clst_rcc,ldapclient,nss_merge,nss_merge_asap,cvmfs,hpc,fastx2,wd_xcpu,bull-2011 R_HOSTNAME=cluster.rz.rwth-aachen.de R_HWARCH=intel_64 R_HWTYPE=i386 R_HW_HOSTNAME=cluster.rz.RWTH-Aachen.DE R_HW_NPROC=64 R_HW_REALMEM=262144 R_IB=bull R_NAME=cluster R_N_INTEL_MIC= R_OPA= R_OSDIST=CENTOS R_OSDISTMAJVERS=7 R_OSDISTVERS=7.4 R_OSNAME=LINUX R_OSTYPE=LINUX R_OSVERSION=3 R_SERVICES=dialog R_SLURM_CLUSTER= R_SLURM_ROLES= R_SVC_LDAP_MASTER=ldap01.hpc.rz.rwth-aachen.de R_WLM_CLUSTER=rcc R_WLM_ROLES=submit R_WLM_TYPE=lsf R_LOGINHOST=cluster.rz.rwth-aachen.de R_REMOTEHOST=189-152.eduroam.rwth-aachen.de NODENAME=189-152.eduroam.rwth-aachen.de MANPATH=/usr/local_rwth/sw/python/3.6.0/x86_64/share/man:/usr/local_rwth/sw/python/3.6.0/x86_64/extra/man:/opt/MPI/openmpi-1.10.4/linux/intel_16.0.2.181/share/man:/opt/intel/Compiler/16.0/2.181/debugger_2016/en/debugger/gdb-ia/man:/opt/intel/Compiler/16.0/2.181/debugger_2016/en/debugger/gdb-mic/man:/opt/intel/Compiler/16.0/2.181/debugger_2016/en/debugger/gdb-igfx/man:/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/benchmarks/mp_linpack/man:/opt/intel/Compiler/16.0/2.181/rwthlnk/man/common:/usr/local_rwth/modules/man:/opt/lsf/10.1/man:/usr/local_host/man:/usr/local_rwth/man:/usr/man:/usr/share/man:/usr/local/man:/usr/local/share/man R_DELIMITER=------------------------------------------------------------------------------ DELIM_LINE=------------------------------------------------------------------------------ R_LOGFILEDIR=/usr/local_host/logfiles R_INITENVDIR=/usr/local_host/etc ENV=/usr/local_host/etc/kshenv BASH_ENV=/usr/local_host/etc/bashenv PUREOPTIONS= -cache-dir=/work/az916605 PSRC=/rwthfs/rz/SW/HPC/examples DAT_OVERRIDE=/usr/local_host/etc/dat.conf JAVA_TOOL_OPTIONS=-Xmx2048m BINARY_TYPE_HPC= LD_LIBRARY_PATH=/usr/local_rwth/sw/python/3.6.0/x86_64/lib:/usr/local_rwth/sw/python/3.6.0/x86_64/extra/lib:/usr/local_rwth/sw/python/3.6.0/x86_64/extra/CBLAS:/home/az916605/apps//lib:/opt/MPI/openmpi-1.10.4/linux/intel_16.0.2.181/lib:/opt/MPI/openmpi-1.10.4/linux/intel_16.0.2.181/lib/lib32:/opt/intel/mic/myo/lib:/opt/intel/mic/coi/host-linux-release/lib:/opt/intel/Compiler/16.0/2.181/rwthlnk/daal/lib/intel64_lin:/opt/intel/Compiler/16.0/2.181/rwthlnk/daal/lib/ia32_lin:/opt/intel/Compiler/16.0/2.181/rwthlnk/daal/lib/intel64_lin_mic:/opt/intel/Compiler/16.0/2.181/rwthlnk/ipp/lib/intel64_lin:/opt/intel/Compiler/16.0/2.181/rwthlnk/ipp/lib/ia32_lin:/opt/intel/Compiler/16.0/2.181/rwthlnk/ipp/lib/intel64_lin_mic:/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/intel64_lin:/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/ia32_lin:/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/intel64_lin_mic:/opt/intel/Compiler/16.0/2.181/rwthlnk/tbb/lib/intel64_lin/gcc4.4:/opt/intel/Compiler/16.0/2.181/rwthlnk/tbb/lib/ia32_lin/gcc4.4:/opt/intel/Compiler/16.0/2.181/rwthlnk/tbb/lib/intel64_lin_mic:/opt/intel/Compiler/16.0/2.181/rwthlnk/compiler/lib/intel64_lin:/opt/intel/Compiler/16.0/2.181/rwthlnk/compiler/lib/ia32_lin:/opt/intel/Compiler/16.0/2.181/rwthlnk/compiler/lib/intel64_lin_mic:/opt/lsf/10.1/linux3.10-glibc2.17-x86_64/lib LSF_SERVERDIR=/opt/lsf/10.1/linux3.10-glibc2.17-x86_64/etc LSF_BINDIR=/opt/lsf/10.1/linux3.10-glibc2.17-x86_64/bin LSF_LIBDIR=/opt/lsf/10.1/linux3.10-glibc2.17-x86_64/lib LSF_ENVDIR=/opt/lsf/conf FINALDIR=/usr/local_rwth/modules/modulefiles/linux/x86-64 MODULEPATH_modshare=/usr/local_rwth/modules/modulefiles/linux/x86-64/DEVELOP:1:/usr/local_rwth/modules/modulefiles/GLOBAL:1 MODULESHOME=/usr/local_rwth/modules/src MODULEPATH=/usr/local_rwth/modules/modulefiles/linux/x86-64/DEVELOP:/usr/local_rwth/modules/modulefiles/GLOBAL _LMFILESmodshare=/usr/local_rwth/modules/modulefiles/linux/x86-64/DEVELOP/openmpi/1.10.4:1:/usr/local_rwth/modules/modulefiles/linux/x86-64/DEVELOP/python/3.6.0:1:/usr/local_rwth/modules/modulefiles/GLOBAL/DEVELOP:1:/usr/local_rwth/modules/modulefiles/linux/x86-64/DEVELOP/intel/16.0:1 LOADEDMODULES_modshare=intel/16.0:1:openmpi/1.10.4:1:python/3.6.0:1:DEVELOP:1 LMFILES=/usr/local_rwth/modules/modulefiles/GLOBAL/DEVELOP:/usr/local_rwth/modules/modulefiles/linux/x86-64/DEVELOP/intel/16.0:/usr/local_rwth/modules/modulefiles/linux/x86-64/DEVELOP/openmpi/1.10.4:/usr/local_rwth/modules/modulefiles/linux/x86-64/DEVELOP/python/3.6.0 LOADEDMODULES=DEVELOP:intel/16.0:openmpi/1.10.4:python/3.6.0 CPATH=/opt/intel/Compiler/16.0/2.181/rwthlnk/daal/include:/opt/intel/Compiler/16.0/2.181/rwthlnk/ipp/include:/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/include:/opt/intel/Compiler/16.0/2.181/rwthlnk/tbb/include FLAGS_AUTOPAR=-parallel _INTERNAL_FLAGS_MKL_RPATH=MKL INTELARCH=64 FLAGS_FAST=-O3 -ip -xSSE4.2 -axCORE-AVX2,AVX -fp-model fast=2 _LIST_FLAGS_LINKER={-L/opt/intel/Compiler/16.0/2.181/rwthlnk/compiler/lib/intel64_lin -L/opt/intel/Compiler/16.0/2.181/rwthlnk/compiler/lib/ia32_lin -L/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/ia32_lin -L/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/intel64_lin } MPM_LAUNCHER=/opt/intel/Compiler/16.0/2.181/debugger_2016/mpm/bin/start_mpm.sh FLAGS_MKL_INCLUDE=-I/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/include -I/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/include/fftw -I/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/include/intel64/lp64 -I/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/include/ia32 -I/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/include/intel64/lp64 _LAST_COMPILER_REVISION=2.181 FLAGS_OPENMP=-qopenmp TBBROOT=/opt/intel/Compiler/16.0/2.181/rwthlnk/tbb FLAGS_ARCH64= -m64 _LISTINTERNAL_MKLROOT=intel INTEL_LICENSE_FILE=50017@license2.rz.rwth-aachen.de _INTERNAL_INTELARCH=intel _INTERNAL_FLAGS_MKL_INCLUDE=MKL OBJCXX= _LIST_FLAGS_FAST_NO_FPOPT={-O3 -ip -xSSE4.2 -axCORE-AVX2,AVX -fp-model precise} FLAGS_LINKER=-L/opt/intel/Compiler/16.0/2.181/rwthlnk/compiler/lib/intel64_lin -L/opt/intel/Compiler/16.0/2.181/rwthlnk/compiler/lib/ia32_lin -L/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/ia32_lin -L/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/intel64_lin MIC_LIBRARY_PATH=/opt/intel/Compiler/16.0/2.181/rwthlnk/daal/lib/intel64_lin_mic:/opt/intel/Compiler/16.0/2.181/rwthlnk/ipp/lib/intel64_lin_mic:/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/intel64_lin_mic:/opt/intel/Compiler/16.0/2.181/rwthlnk/tbb/lib/intel64_lin_mic:/opt/intel/Compiler/16.0/2.181/rwthlnk/compiler/lib/intel64_lin_mic FLAGS_INTEL_FREE=-free -fpp _LISTLAST_COMPILER_MINORVERSION=0 FLAGS_INTEL_ARCH32= -m32 _LIST_FLAGS_AUTOPAR=-parallel _LIST__INTERNAL_FLAGS_MKL_RPATH=MKL MIC_LIBRARY_PATH_modshare=/opt/intel/Compiler/16.0/2.181/rwthlnk/compiler/lib/intel64_lin_mic:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/intel64_lin_mic:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/tbb/lib/intel64_lin_mic:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/ipp/lib/intel64_lin_mic:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/daal/lib/intel64_lin_mic:1 _LIST_INTELARCH=64 F77=ifort -f77rtl -fpconstant -intconstant _LIST_FLAGS_MKL_INCLUDE={-I/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/include -I/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/include/fftw -I/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/include/intel64/lp64 -I/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/include/ia32 -I/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/include/intel64/lp64} _LIST_CXX=icpc _LIST_FLAGS_DEBUG=-g _INTERNAL_FLAGS_MKL_LINKER=MKL _LIST_MKLROOT=/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl MANPATH_modshare=/usr/local_rwth/sw/python/3.6.0/x86_64/share/man:1:/usr/local/share/man:1:/opt/lsf/10.1/man:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/man/common:1:/opt/intel/Compiler/16.0/2.181/debugger_2016/en/debugger/gdb-igfx/man:1:/usr/local_rwth/sw/python/3.6.0/x86_64/extra/man:1:/usr/local_rwth/man:1:/usr/local_host/man:1:/opt/intel/Compiler/16.0/2.181/debugger_2016/en/debugger/gdb-mic/man:1:/usr/man:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/benchmarks/mp_linpack/man:1:/usr/local/man:1:/opt/MPI/openmpi-1.10.4/linux/intel_16.0.2.181/share/man:1:/usr/local_rwth/modules/man:1:/opt/intel/Compiler/16.0/2.181/debugger_2016/en/debugger/gdb-ia/man:1:/usr/share/man:1 _LIST_FC=ifort INCLUDE=/opt/intel/Compiler/16.0/2.181/rwthlnk/daal/include:/opt/intel/Compiler/16.0/2.181/rwthlnk/ipp/include:/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/include:/opt/intel/Compiler/16.0/2.181/rwthlnk/tbb/include prepvintFLAGS_LPATH={-L/opt/intel/Compiler/16.0/2.181/rwthlnk/compiler/lib/intel64_lin -L/opt/intel/Compiler/16.0/2.181/rwthlnk/compiler/lib/ia32_lin -L/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/intel64_lin -L/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/ia32_lin } FLAGS_FREE=-free -fpp LAST_COMPILER=intel _INTERNAL_FLAGS_LPATH=intel _LIST_FLAGS_MKL_RPATH={ -Wl,-rpath=/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/intel64_lin -Wl,-rpath=/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/ia32_lin -Wl,-rpath=/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/intel64_lin_mic} MKL_MIC_ENABLE=0 FLAGS_MATH_LINKER=-mkl=sequential LD_LIBRARY_PATH_modshare=/usr/local_rwth/sw/python/3.6.0/x86_64/lib:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/ipp/lib/intel64_lin:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/daal/lib/intel64_lin:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/ipp/lib/ia32_lin:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/compiler/lib/intel64_lin:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/compiler/lib/ia32_lin:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/intel64_lin:2:/opt/lsf/10.1/linux3.10-glibc2.17-x86_64/lib:1:/opt/intel/mic/myo/lib:1:/opt/MPI/openmpi-1.10.4/linux/intel_16.0.2.181/lib:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/tbb/lib/intel64_lin_mic:1:/usr/local_rwth/sw/python/3.6.0/x86_64/extra/CBLAS:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/ipp/lib/intel64_lin_mic:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/daal/lib/intel64_lin_mic:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/tbb/lib/intel64_lin/gcc4.4:1:/usr/local_rwth/sw/python/3.6.0/x86_64/extra/lib:1:/opt/MPI/openmpi-1.10.4/linux/intel_16.0.2.181/lib/lib32:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/daal/lib/ia32_lin:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/compiler/lib/intel64_lin_mic:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/tbb/lib/ia32_lin/gcc4.4:1:/opt/intel/mic/coi/host-linux-release/lib:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/intel64_lin_mic:2:/home/az916605/apps//lib:999999999:/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/ia32_lin:2 _LISTINTERNAL_INTEL_LICENSE_FILE=intel prepvintFLAGS_RPATH={djkch84adca/opt/intel/Compiler/16.0/2.181/rwthlnk/compiler/lib/intel64_lin djkch84adca/opt/intel/Compiler/16.0/2.181/rwthlnk/compiler/lib/ia32_lin djkch84adca/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/intel64_lin djkch84adca/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/ia32_lin } _LIST_FLAGS_FAST={-O3 -ip -xSSE4.2 -axCORE-AVX2,AVX -fp-model fast=2 } _LISTLAST_COMPILER_MAJORVERSION=16 OMP_NUM_THREADS=1 _INTERNAL_FLAGS_RPATH=intel INTEL_PYTHONHOME=/opt/intel/Compiler/16.0/2.181/debugger_2016/python/intel64/ _LIST_FLAGS_MKL_LINKER=-mkl=sequential CC=icc FLAGS_FAST_NO_FPOPT=-O3 -ip -xSSE4.2 -axCORE-AVX2,AVX -fp-model precise FLAGS_INTEL_RUNTIME_LINKER=-Wl,-rpath= _LAST_COMPILER_MINORVERSION=0 _LIST_INTEL_LICENSE_FILE=50017@license2.rz.rwth-aachen.de FLAGS_PIC=-fPIC -shared _INTERNAL_INTEL_LICENSE_FILE=intel _LIST_LAST_COMPILER=intel _LIST_OBJCXX={ } _LIST_FLAGS_ARCH32={ -m32 } INTEL_CC=icc _LISTINTERNAL_FLAGS_LPATH=intel _LAST_REPLACES_FLAGS_RPATH=-Wl,-rpath= CXX=icpc FLAGS_DEBUG=-g FLAGS_MKL_LINKER=-mkl=sequential FLAGS_LPATH=-L/opt/intel/Compiler/16.0/2.181/rwthlnk/compiler/lib/intel64_lin -L/opt/intel/Compiler/16.0/2.181/rwthlnk/compiler/lib/ia32_lin -L/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/intel64_lin -L/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/ia32_lin
GDB_CROSS=/opt/intel/Compiler/16.0/2.181/debugger_2016/gdb/intel64_mic/bin/gdb-mic DAALROOT=/opt/intel/Compiler/16.0/2.181/rwthlnk/daal FLAGS_INTEL_OPENMP=-qopenmp FLAGS_INTEL_ARCH64= -m64 FLAGS_ARCH32= -m32 INTEL_F77=ifort -f77rtl -fpconstant -intconstant _LISTINTERNAL_FLAGS_RPATH=intel MKLROOT=/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl _LISTINTERNAL_FLAGS_MKL_LINKER=MKL FLAGS_MATH_INCLUDE=-I/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/include -I/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/include/fftw -I/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/include/intel64/lp64 -I/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/include/ia32 -I/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/include/intel64/lp64 INTEL_OBJC= INTEL_OBJCXX= FLAGS_INTEL_FAST_NO_FPOPT=-O3 -ip -xSSE4.2 -axCORE-AVX2,AVX -fp-model precise NLSPATH=/opt/intel/Compiler/16.0/2.181/debugger2016/gdb/intel64/share/locale/%l%t/%N:/opt/intel/Compiler/16.0/2.181/debugger_2016/gdb/intel64mic/share/locale/%l%t/%N:/opt/intel/Compiler/16.0/2.181/rwthlnk/daal/lib/intel64lin/locale/%l%t/%N:/opt/intel/Compiler/16.0/2.181/rwthlnk/ipp/lib/intel64lin/locale/%l%t/%N:/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/intel64lin/locale/%l%t/%N:/opt/intel/Compiler/16.0/2.181/rwthlnk/compiler/lib/intel64lin/locale/%l%t/%N _LIST_FLAGS_PIC={-fPIC -shared} FLAGS_RPATH=-Wl,-rpath=/opt/intel/Compiler/16.0/2.181/rwthlnk/compiler/lib/intel64_lin -Wl,-rpath=/opt/intel/Compiler/16.0/2.181/rwthlnk/compiler/lib/ia32_lin -Wl,-rpath=/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/intel64_lin -Wl,-rpath=/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/ia32_lin FLAGS_INTEL_LINKER=-L/opt/intel/Compiler/16.0/2.181/rwthlnk/compiler/lib/intel64_lin -L/opt/intel/Compiler/16.0/2.181/rwthlnk/compiler/lib/ia32_lin -L/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/ia32_lin -L/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/intel64_lin _LISTLAST_COMPILER_REVISION=2.181 FLAGS_MKL_RPATH= -Wl,-rpath=/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/intel64_lin -Wl,-rpath=/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/ia32_lin -Wl,-rpath=/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/intel64_lin_mic INCLUDE_modshare=/opt/intel/Compiler/16.0/2.181/rwthlnk/ipp/include:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/daal/include:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/include:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/tbb/include:1 FLAGS_INTEL_DEBUG=-g _LISTINTERNAL_INTELARCH=intel _LIST__INTERNAL_FLAGS_MKL_INCLUDE=MKL _LIST_FLAGS_FREE={-free -fpp} _LIST_OBJC={ } INFOPATH=/opt/intel/Compiler/16.0/2.181/debugger_2016/en/debugger/gdb-ia/info:/opt/intel/Compiler/16.0/2.181/debugger_2016/en/debugger/gdb-mic/info:/opt/intel/Compiler/16.0/2.181/debugger_2016/en/debugger/gdb-igfx/info LIBRARY_PATH=/opt/intel/Compiler/16.0/2.181/rwthlnk/daal/lib/intel64_lin:/opt/intel/Compiler/16.0/2.181/rwthlnk/daal/lib/ia32_lin:/opt/intel/Compiler/16.0/2.181/rwthlnk/daal/lib/intel64_lin_mic:/opt/intel/Compiler/16.0/2.181/rwthlnk/ipp/lib/intel64_lin:/opt/intel/Compiler/16.0/2.181/rwthlnk/ipp/lib/ia32_lin:/opt/intel/Compiler/16.0/2.181/rwthlnk/ipp/lib/intel64_lin_mic:/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/intel64_lin:/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/ia32_lin:/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/intel64_lin_mic:/opt/intel/Compiler/16.0/2.181/rwthlnk/tbb/lib/intel64_lin/gcc4.4:/opt/intel/Compiler/16.0/2.181/rwthlnk/tbb/lib/ia32_lin/gcc4.4:/opt/intel/Compiler/16.0/2.181/rwthlnk/tbb/lib/intel64_lin_mic MIC_LD_LIBRARY_PATH_modshare=/opt/intel/Compiler/16.0/2.181/rwthlnk/compiler/lib/intel64_lin_mic:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/intel64_lin_mic:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/tbb/lib/intel64_lin_mic:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/ipp/lib/intel64_lin_mic:1:/opt/intel/mic/myo/lib:1:/opt/intel/mic/coi/device-linux-release/lib:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/daal/lib/intel64_lin_mic:1 INFOPATH_modshare=/opt/intel/Compiler/16.0/2.181/debugger_2016/en/debugger/gdb-ia/info:1:/opt/intel/Compiler/16.0/2.181/debugger_2016/en/debugger/gdb-igfx/info:1:/opt/intel/Compiler/16.0/2.181/debugger_2016/en/debugger/gdb-mic/info:1 LIBRARY_PATH_modshare=/opt/intel/Compiler/16.0/2.181/rwthlnk/ipp/lib/intel64_lin:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/ipp/lib/ia32_lin:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/daal/lib/intel64_lin:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/intel64_lin:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/tbb/lib/intel64_lin_mic:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/ipp/lib/intel64_lin_mic:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/tbb/lib/intel64_lin/gcc4.4:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/daal/lib/intel64_lin_mic:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/tbb/lib/ia32_lin/gcc4.4:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/daal/lib/ia32_lin:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/intel64_lin_mic:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/ia32_lin:1 _LAST_COMPILER_MAJORVERSION=16 FLAGS_MATH_RPATH= -Wl,-rpath=/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/intel64_lin -Wl,-rpath=/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/ia32_lin -Wl,-rpath=/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/intel64_lin_mic NLSPATH_modshare=/opt/intel/Compiler/16.0/2.181/debugger2016/gdb/intel64/share/locale/%l%t/%N:1:/opt/intel/Compiler/16.0/2.181/debugger_2016/gdb/intel64mic/share/locale/%l%t/%N:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/daal/lib/intel64lin/locale/%l%t/%N:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/compiler/lib/intel64lin/locale/%l%t/%N:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/intel64lin/locale/%l%t/%N:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/ipp/lib/intel64lin/locale/%l%t/%N:1 IPPROOT=/opt/intel/Compiler/16.0/2.181/rwthlnk/ipp FLAGS_INTEL_FAST=-O3 -ip -xSSE4.2 -axCORE-AVX2,AVX -fp-model fast=2 _LIST_FLAGS_RUNTIME_LINKER=-Wl,-rpath= _LIST_F77={ifort -f77rtl -fpconstant -intconstant} OBJC= MIC_LD_LIBRARY_PATH=/opt/intel/mic/myo/lib:/opt/intel/mic/coi/device-linux-release/lib:/opt/intel/Compiler/16.0/2.181/rwthlnk/daal/lib/intel64_lin_mic:/opt/intel/Compiler/16.0/2.181/rwthlnk/ipp/lib/intel64_lin_mic:/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/lib/intel64_lin_mic:/opt/intel/Compiler/16.0/2.181/rwthlnk/tbb/lib/intel64_lin_mic:/opt/intel/Compiler/16.0/2.181/rwthlnk/compiler/lib/intel64_lin_mic _LIST_CC=icc FC=ifort CPATH_modshare=/opt/intel/Compiler/16.0/2.181/rwthlnk/ipp/include:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/daal/include:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/mkl/include:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/tbb/include:1 _INTERNAL_MKLROOT=intel FLAGS_INTEL_PIC=-fPIC -shared FLAGS_INTEL_AUTOPAR=-parallel FLAGS_RUNTIME_LINKER=-Wl,-rpath= INTEL_FC=ifort _LIST_FLAGS_OPENMP=-qopenmp _LIST_FLAGS_ARCH64={ -m64 } INTEL_CXX=icpc PATH_modshare=/usr/local_host/bin:1:/opt/intel/Compiler/16.0/2.181/debugger_2016/gdb/intel64/bin:1:/usr/bin:1:/opt/intel/Compiler/16.0/2.181/rwthlnk/bin/intel64:1:/usr/local_rwth/sw/python/3.6.0/x86_64/bin:1:/opt/lsf/10.1/linux3.10-glibc2.17-x86_64/etc:1:/usr/local_host/sbin:1:/usr/local/bin:1:/opt/lsf/10.1/linux3.10-glibc2.17-x86_64/bin:1:/bin:1:/opt/intel/Compiler/16.0/2.181/debugger_2016/gdb/intel64_mic/bin:1:/usr/local_rwth/sbin:1:/opt/MPI/openmpi-1.10.4/linux/intel_16.0.2.181/bin:1:/sbin:1:/usr/sbin:1:/usr/local_rwth/sw/python/3.6.0/x86_64/extra/bin:1:.:1:/usr/local/sbin:1:/opt/MPI/bin:1:/usr/local_rwth/bin:1:/home/az916605/apps//bin:999999999 OMPI_MCA_mtl=^psm MPI_ROOT=/opt/MPI/openmpi-1.10.4/linux/intel_16.0.2.181 MPI_INCLUDE=/opt/MPI/openmpi-1.10.4/linux/intel_16.0.2.181/include OMPI_MCA_btl_openib_ib_timeout=24 MPIF77=mpif77 _LAST_MPI_MINORVERSION=10 OMPI_MCA_oob_tcp_if_include=ib0 MPI_LIBDIR=/opt/MPI/openmpi-1.10.4/linux/intel_16.0.2.181/lib OMPI_MCA_hwloc_base_binding_policy=none MPICC=mpicc OMPI_MCA_btl_openib_warn_default_gid_prefix=0 OMPI_MCA_carto=file MPI_BINDIR=/opt/MPI/openmpi-1.10.4/linux/intel_16.0.2.181/bin OMPI_MCA_btl=^tcp _LAST_MPI_MAJORVERSION=1 _LAST_MPI_REVISION=4 OMPI_MCA_carto_file_path=/opt/MPI/openmpi/carto/carto_2K-64C.txt _LAST_MPI=openmpi MPICXX=mpicxx _COMPILER_OPENMPI=intel OMPI_MCA_hwloc_base_report_bindings=0 FLAGS_MPI_BATCH= -np 2 MPIEXEC=/opt/MPI/bin/mpiexec -mca oob_tcp_if_include ib0 -mca btl_tcp_if_include ib0 OMPI_MCA_btl_tcp_if_include=ib0 _MPI_LOADED=openmpi/1.10.4 FLAGS_MKL_SCALAPACK_LINKER=-lmkl_scalapack_lp64 -lmkl_blacs_openmpi_lp64 -lpthread -lm MPITEST=/opt/MPI/bin/mpitest MPI_NAME=openmpi MPIFC=mpif90 R_SHLVL_INTERACTIVE=1 EDITOR=vi VISUAL=vi FCEDIT=vi PAGER=less APPS_ROOT=/home/az916605/apps/ HDF5_DIR=/home/az916605/apps//hdf5-1.10.0-patch1-linux-centos7-x86_64-gcc485-shared/ PYTHONPATH=/usr/local_rwth/sw/python/3.6.0/x86_64/extra/lib/python3.6/site-packages PYTHONPATH_modshare=/usr/local_rwth/sw/python/3.6.0/x8664/extra/lib/python3.6/site-packages:1 =/usr/bin/printenv

miklos1 commented 6 years ago

Can you try unsetting CC and CXX?

agzimmerman commented 6 years ago

I did

env -u PYTHONPATH -u CC -u CXX python3 firedrake-install --no-package-manager

and now, while its building PETSc, I see

Catastrophic error: could not set locale "" to allow processing of multibyte characters

which, based on some quick Googling, I guess means that my environment variable LANG=C.UTF-8 might need to be changed. I'll have to try another build, though I don't yet know why I should have to change LANG and what I should change it to.

Since firedrake-install requires me to always start from scratch, and especially since the problems are occurring while building PETSc, these iterations take a while.

This seems like a new issue, and it's not entirely clear yet whether or not unsetting CC and CXX has helped the other issue; but since I've hit a new issue, perhaps that did help.

Edit: Or perhaps I leave LANG alone and need to add LC_ALL=C.UTF-8.

wence- commented 6 years ago

If we're failing to build PETSc, it is possible to build that separately and then point the firedrake installer to it. You just have to make sure you get a matching version. This speeds up the debug cycle.

You can do:

git clone https://github.com/firedrakeproject/petsc.git
cd petsc
./configure --with-shared-libraries=1 --with-debugging=0 --with-c2html=0 --download-chaco --download-metis --download-parmetis --download-scalapack --download-hypre --download-mumps --download-netcdf --download-hdf5 --download-pnetcdf --download-exodusii

And then follow the instructions for building.

This gets you a PETSc that matches the one that firedrake-install would build.

You then have to remember to set PETSC_DIR and PETSC_ARCH appropriately, and run firedrake-install --honour-petsc-dir.

agzimmerman commented 6 years ago

@wence- , thanks for explaining how I can speed up the debugging process.

@miklos1, now I'm back to a mpicc related error. I tried

LC_ALL=C.UTF-8 env -u PYTHONPATH -u CC -u CXX python3 firedrake-install --no-package-manager

and now here's an excerpt from firedrake-install.log:

TESTING: checkCCompiler from config.setCompilers(/w0/tmp/az916605/cluster_18312/pip-req-build -9zf_mqez/config/BuildSystem/config/setCompilers.py:586)

         UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
-------------------------------------------------------------------------------
C compiler you provided with -with-cc=mpicc does not work.
Cannot compile C with mpicc.
*******************************************************************************

Unfortunately I cannot find configure.log which supposedly has details.

How I should proceed?

wence- commented 6 years ago

OK, so the problem is somehow that mpicc does not work. Because this is a problem building PETSc, we need to debug this with configure.log. Unfortunately, the pip installation procedure for PETSc hides all this.

Therefore I suggest attempting to build PETSc "standalone" (following the instructions above), and then we will see what the issue is.

wence- commented 5 years ago

I hope this was fixed, FWIW, we are now better at propagating the compiler environment into builds inside firedrake-install.

agzimmerman commented 5 years ago

Thanks for the update. I gave up on trying to build this on CentOS and also our IT didn't have a good time with it; but now we have Singularity and just in the last days I ran my code using Firedrake on that successfully for the first time.