imkl needs -fPIC added for MPI bits (e.g. mpi_transpose) in FFTW3(2?) wrappers

[JackPerdue]

I'm trying to build libMesh [1] with the intel/2016a toolchain and hitting this problem:

ld: /software/easybuild/software/imkl/11.3.1.150-iimpi-8.1.5-GCC-4.9.3-2.25/mkl/lib/intel64/libfftw3x_cdft_lp64_pic.a(mpi_transpose.o): relocation R_X86_64_32 against .rodata.str1.4' can not be used when making a shared object; recompile with -fPIC /software/easybuild/software/imkl/11.3.1.150-iimpi-8.1.5-GCC-4.9.3-2.25/mkl/lib/intel64/libfftw3x_cdft_lp64_pic.a: error adding symbols: Bad value make[2]: *** [libmesh_dbg.la] Error 1 make[2]: Leaving directory/tmp/easybuild/libMesh/1.0.0/intel-2016a-Python-2.7.11/libmesh-1.0.0' make[1]: *\ [../libmesh_dbg.la] Error 2

AFAICT, mpi_transpose truly was not compiled with -fPIC despite it being in the _pic library.

mpi_transpose seems to be fairly new (11.3-ish).

Anyone got a quick fix?

[1] https://github.com/libMesh/libmesh/

[JackPerdue]

# Authors:: Jack Perdue <j-perdue@tamu.edu> - TAMU HPRC - http://hprc.tamu.edu

easyblock = 'ConfigureMake'

name = 'libMesh'
version = '1.0.0'
versionsuffix = '-Python-%(pyver)s'

homepage = 'https://github.com/google/glog'
description = """
 C++ implementation of the Google logging module
"""

toolchain = {'name': 'intel', 'version': '2016a'}
toolchainopts = {'pic': True, 'opt': True, 'optarch': True}

dependencies = [
    ('Python', '2.7.11'),  # se we can use pyver above
    #('Boost', '1.61.0', versionsuffix),  # loads bzip2, zlib, Python
    ('CppUnit', '1.12.1'),  # no deps
    #('cURL', '7.47.0'),  # no deps
    ('Eigen', '3.2.8'),  # no deps
    #('HDF5', '1.8.16'),  # loads zlib and Szip
    ('netCDF', '4.4.0'),  # loads HDF5 and cURL
    #('ParMETIS', '4.0.3'),  # no deps (???)
    ('PETSc', '3.7.2', versionsuffix),  # loads Boost, ScientificPython, FIAT, METIS, ParMETIS, SCOTCH, SuiteSparse, Hypre
    ('Qhull', '2015.2'),  # no deps
    ('tbb', '4.4.2.152', '', True),  # nodeps
    ('VTK', '6.3.0', versionsuffix),  # loads Python and libGLU
]

# https://github.com/libMesh/libmesh/archive/v1.0.0.tar.gz
source_urls = ['https://github.com/libMesh/libmesh/archive']
sources = [SOURCELOWER_TAR_GZ]

preconfigopts = 'ml -t && '

configopts = """\
    --with-methods="opt oprof dbg"\
    --enable-silent-rules\
    --enable-unique-id\
    --disable-warnings\
    --enable-unique-ptr\
    --enable-openmp\
    --disable-maintainer-mode\
    --disable-netcdf\
    --enable-curl\
    --with-tbb=$EBROOTTBB\
    --with-metis=PETSc\
    --with-vtk-include=$EBROOTVTK/include/vtk-6.3\
    --with-vtk-lib=$EBROOTVTK/lib\
    --with-eigen-include=$EBROOTEIGEN/include\
    --with-hdf5=$EBROOTHDF5\
    --with-cppunit-prefix=$EBROOTCPPUNIT\
    --with-curl-include=$EBROOTCURL/include/curl\
    --with-curl-lib=$EBROOTCURL/lib\
"""
# --with-trilinos=$EBROOTTRILINOS\

prebuildopts = 'sed -e "s/--silent//" -i.eb Makefile && '

sanity_check_paths = {
    'dirs': ['include', 'lib', 'share'],
    'files': ['include/glog/logging.h'],
}

moduleclass = 'tools'
# EOF

[JackPerdue]

[obviously a work in progress (I copied my recent glog config)]

[boegel]

@JackPerdue: the imkl easyblock does intend to pass -fPIC for the _pic libraries, but maybe this got broken somehow for recent versions of Intel MKL without us knowing it...

See the last part of post_install_step in the imkl easyblock (the bit guarded by if self.cfg['interfaces']:...

Can you look into providing the relevant part of a debug log (look for make -f makefile commands being executed)?

[JackPerdue]

This appears to be Intel's fault, not EB. 11.3.1 and 11.3.2 use CFLAGS= instead of CFLAGS+=. Seems to be fixed 11.3.3.

Here's my fix for now:

postinstallcmds = [
    'sed -e "s/CFLAGS = /CFLAGS += /" -i.eb %(installdir)s/compilers_and_libraries_2016.1.150/linux/mkl/interfaces/fftw3x_cdft/makefile',
]

[JackPerdue]

The above change and rebuild seemed to be fairly painless on a live system (nobody screamed). Solved my problem. YMMV. (it is a rather key component for those using FFTW w/intel/2016a).

Here's the completed (now built) libMesh-1.0.0. On to kill a MOOSE (this one: http://www.mooseframework.org/ ....not this one... https://moose.ncbs.res.in/ .... gonna have to work on my aim).

# Authors:: Jack Perdue  - TAMU HPRC - http://hprc.tamu.edu
easyblock = 'ConfigureMake'
name = 'libMesh'
version = '1.0.0'
versionsuffix = '-Python-%(pyver)s'
homepage = 'http://libmesh.github.io/'
description = """
 The libMesh library provides a framework for the numerical simulation of
 partial differential equations using arbitrary unstructured discretizations
 on serial and parallel platforms. A major goal of the library is to provide
 support for adaptive mesh refinement (AMR) computations in parallel while
 allowing a research scientist to focus on the physics they are modeling.
"""
toolchain = {'name': 'intel', 'version': '2016a'}
toolchainopts = {'pic': True, 'opt': True, 'optarch': True}
dependencies = [
    #('Boost', '1.61.0', versionsuffix),  # loads bzip2, zlib, Python
    ('CppUnit', '1.12.1'),  # no deps
    #('cURL', '7.47.0'),  # no deps
    ('Eigen', '3.2.8'),  # no deps
    #('HDF5', '1.8.16'),  # loads zlib and Szip
    ('netCDF', '4.4.0'),  # loads HDF5 and cURL
    #('ParMETIS', '4.0.3'),  # no deps (???)
    ('PETSc', '3.7.2', versionsuffix),  # loads Boost, ScientificPython, FIAT, METIS, ParMETIS, SCOTCH, SuiteSparse, Hypre
    ('Python', '2.7.11'),  # so we can use pyver in versionsuffix
    ('Qhull', '2015.2'),  # no deps
    ('tbb', '4.4.2.152', '', True),  # nodeps
    ('VTK', '6.3.0', versionsuffix),  # loads Python and libGLU
]
# https://github.com/libMesh/libmesh/archive/v1.0.0.tar.gz
source_urls = ['https://github.com/libMesh/libmesh/archive']
sources = [SOURCELOWER_TAR_GZ]
# NOTE: Settings here come from MOOSE's update_and_rebuild_libmesh.sh script
configopts = """\
    --with-methods="opt oprof dbg"\
    --enable-silent-rules\
    --enable-unique-id\
    --disable-warnings\
    --enable-unique-ptr\
    --enable-openmp\
    --disable-maintainer-mode\
    --disable-netcdf\
    --enable-curl\
    --with-tbb=$EBROOTTBB\
    --with-metis=PETSc\
    --with-vtk-include=$EBROOTVTK/include/vtk-6.3\
    --with-vtk-lib=$EBROOTVTK/lib\
    --with-eigen-include=$EBROOTEIGEN/include\
    --with-hdf5=$EBROOTHDF5\
    --with-cppunit-prefix=$EBROOTCPPUNIT\
    --with-curl-include=$EBROOTCURL/include/curl\
    --with-curl-lib=$EBROOTCURL/lib\
"""
# --with-trilinos=$EBROOTTRILINOS\
prebuildopts = """
    sed -e "s/--silent//" -i.eb1 Makefile
    sed -e "s/--silent//" -i.eb1 tests/Makefile
    sed -e "s/-lcholmod/-lcholmod -lmetis/" -i.eb2 Makefile
    sed -e "s/-lcholmod/-lcholmod -lmetis/" -i.eb2 tests/Makefile
"""
sanity_check_paths = {
    'dirs':  ['bin', 'etc/libmesh', 'include/libmesh', 'lib/pkgconfig',
              'share'],
    'files': ['bin/libmesh-config', 'etc/libmesh/libmesh.pc',
              'include/libmesh/libmesh_base.h', 'lib/libmesh_opt.la'],
}
moduleclass = 'math'
# EOF

[boegel]

@JackPerdue I guess it makes sense to incorporate the patching of fftw3x_cdft/makefile you do in the imkl easyblock (with a proper version check in place)?

[boegel]

I was bitten by this too recently, so I applied the fix @JackPerdue came up with consistently for the 11.3.1 and 11.3.2 imkl versions, cfr. https://github.com/hpcugent/easybuild-easyconfigs/pull/4198