Trouble creating/building GDAL-2.2.3-intel-2018b-Python-2.7.15.eb

[casparvl]

Issue I have created a GDAL-2.2.3-intel-2018b-Python-2.7.15.eb (included at the bottom, for reference).

On on of our two systems, building this fails with:

icpc -shared -L/hpc/sw_test/EB_regression/2019/software/icc/2018.3.222-GCC-7.3.0-2.30/lib/intel64 -L/hpc/sw_test/EB_regression/2019/software/imkl/2018.3.222-iimpi-2018b/lib -L/hpc/sw_test/EB_regression/2019/software/imkl/2018.3.222-iimpi-2018b/mkl/lib/intel64 -L/hpc/sw_test/EB_regression/2019/software/imkl/2018.3.222-iimpi-2018b/lib -L/hpc/sw_test/EB_regression/2019/software/bzip2/1.0.6-GCCcore-7.3.0/lib -L/hpc/sw_test/EB_regression/2019/software/zlib/1.2.11-GCCcore-7.3.0/lib -L/hpc/sw_test/EB_regression/2019/software/libreadline/7.0-GCCcore-7.3.0/lib -L/hpc/sw_test/EB_regression/2019/software/ncurses/6.1-GCCcore-7.3.0/lib -L/hpc/sw_test/EB_regression/2019/software/SQLite/3.24.0-GCCcore-7.3.0/lib -L/hpc/sw_test/EB_regression/2019/software/GMP/6.1.2-GCCcore-7.3.0/lib -L/hpc/sw_test/EB_regression/2019/software/libffi/3.2.1-GCCcore-7.3.0/lib64 -L/hpc/sw_test/EB_regression/2019/software/libffi/3.2.1-GCCcore-7.3.0/lib -L/hpc/sw_test/EB_regression/2019/software/icc/2018.3.222-GCC-7.3.0-2.30/lib/intel64 -L/hpc/sw_test/EB_regression/2019/software/imkl/2018.3.222-iimpi-2018b/lib -L/hpc/sw_test/EB_regression/2019/software/imkl/2018.3.222-iimpi-2018b/mkl/lib/intel64 -L/hpc/sw_test/EB_regression/2019/software/imkl/2018.3.222-iimpi-2018b/lib -L/hpc/sw_test/EB_regression/2019/software/Python/2.7.15-intel-2018b/lib -L/hpc/sw_test/EB_regression/2019/software/netCDF/4.6.1-intel-2018b/lib64 -L/hpc/sw_test/EB_regression/2019/software/expat/2.2.5-GCCcore-7.3.0/lib -L/home/casparl/.local/EB_regression/RedHatEnterpriseServer7/software/GEOS/3.6.2-intel-2018b-Python-2.7.15/lib -L/hpc/sw_test/EB_regression/2019/software/SQLite/3.24.0-GCCcore-7.3.0/lib -L/hpc/sw_test/EB_regression/2019/software/libxml2/2.9.8-GCCcore-7.3.0/lib -L/hpc/sw_test/EB_regression/2019/software/libpng/1.6.34-GCCcore-7.3.0/lib -L/hpc/sw_test/EB_regression/2019/software/libjpeg-turbo/2.0.0-GCCcore-7.3.0/lib64 -L/home/casparl/.local/EB_regression/RedHatEnterpriseServer7/software/JasPer/2.0.14-GCCcore-7.3.0/lib64 -L/hpc/sw_test/EB_regression/2019/software/LibTIFF/4.0.9-GCCcore-7.3.0/lib -L/hpc/sw_test/EB_regression/2019/software/zlib/1.2.11-GCCcore-7.3.0/lib -L/hpc/sw_test/EB_regression/2019/software/cURL/7.60.0-GCCcore-7.3.0/lib -L/hpc/sw_test/EB_regression/2019/software/PCRE/8.41-GCCcore-7.3.0/lib -L/home/casparl/.local/EB_regression/RedHatEnterpriseServer7/software/PROJ/5.0.0-intel-2018b/lib -DHAVE_AVX_AT_COMPILE_TIME -DHAVE_SSSE3_AT_COMPILE_TIME -DHAVE_SSE_AT_COMPILE_TIME -O2 -O2 -xAVX -axCORE-AVX2,CORE-AVX512 -ftz -fp-speculation=safe -fp-model source -Wall -Wdeclaration-after-statement -diag-disable 188,1684,2259,2304,3280,11074,11076 -Wextra -Winit-self -Wunused-parameter -Wmissing-prototypes -Wmissing-declarations -Wformat -Wshorten-64-to-32 -Wshadow -Werror=vla -Wdeclaration-after-statement -DGNM_ENABLED -I/hpc/sw_test/EB_regression/2019/software/imkl/2018.3.222-iimpi-2018b/mkl/include -I/hpc/sw_test/EB_regression/2019/software/Python/2.7.15-intel-2018b/include -I/hpc/sw_test/EB_regression/2019/software/netCDF/4.6.1-intel-2018b/include -I/hpc/sw_test/EB_regression/2019/software/expat/2.2.5-GCCcore-7.3.0/include -I/home/casparl/.local/EB_regression/RedHatEnterpriseServer7/software/GEOS/3.6.2-intel-2018b-Python-2.7.15/include -I/hpc/sw_test/EB_regression/2019/software/SQLite/3.24.0-GCCcore-7.3.0/include -I/hpc/sw_test/EB_regression/2019/software/libxml2/2.9.8-GCCcore-7.3.0/include -I/hpc/sw_test/EB_regression/2019/software/libpng/1.6.34-GCCcore-7.3.0/include -I/hpc/sw_test/EB_regression/2019/software/libjpeg-turbo/2.0.0-GCCcore-7.3.0/include -I/home/casparl/.local/EB_regression/RedHatEnterpriseServer7/software/JasPer/2.0.14-GCCcore-7.3.0/include -I/hpc/sw_test/EB_regression/2019/software/LibTIFF/4.0.9-GCCcore-7.3.0/include -I/hpc/sw_test/EB_regression/2019/software/zlib/1.2.11-GCCcore-7.3.0/include -I/hpc/sw_test/EB_regression/2019/software/cURL/7.60.0-GCCcore-7.3.0/include -I/hpc/sw_test/EB_regression/2019/software/PCRE/8.41-GCCcore-7.3.0/include -I/home/casparl/.local/EB_regression/RedHatEnterpriseServer7/software/PROJ/5.0.0-intel-2018b/include -I/tmp/casparl/GDAL/2.2.3/intel-2018b-Python-2.7.15/gdal-2.2.3/port -I/home/casparl/.local/EB_regression/RedHatEnterpriseServer7/software/JasPer/2.0.14-GCCcore-7.3.0 -I/home/casparl/.local/EB_regression/RedHatEnterpriseServer7/software/JasPer/2.0.14-GCCcore-7.3.0/include -I/hpc/sw_test/EB_regression/2019/software/netCDF/4.6.1-intel-2018b/include -I/hpc/sw_test/EB_regression/2019/software/libjpeg-turbo/2.0.0-GCCcore-7.3.0 -I/hpc/sw_test/EB_regression/2019/software/libjpeg-turbo/2.0.0-GCCcore-7.3.0/include -I/hpc/sw_test/EB_regression/2019/software/LibTIFF/4.0.9-GCCcore-7.3.0/include -I/hpc/sw_test/EB_regression/2019/software/libpng/1.6.34-GCCcore-7.3.0 -I/hpc/sw_test/EB_regression/2019/software/libpng/1.6.34-GCCcore-7.3.0/include -DGDAL_COMPILATION build/temp.linux-x86_64-2.7/extensions/gdal_wrap.o -L../../.libs -L../../ -L/hpc/sw_test/EB_regression/2019/software/Python/2.7.15-intel-2018b/lib -L/tmp/casparl/GDAL/2.2.3/intel-2018b-Python-2.7.15/gdal-2.2.3/lib -lpython2.7 -lgdal -o build/lib.linux-x86_64-2.7/osgeo/_gdal.so

          ": internal error: ** The compiler has encountered an unexpected problem.
** Segmentation violation signal raised. **
Access violation or stack overflow. Please contact Intel Support for assistance.

icpc: error #10014: problem during multi-file optimization compilation (code 4)
error: command 'icpc' failed with exit status 4
make[2]: *** [build] Error 1
make[2]: Leaving directory `/tmp/casparl/GDAL/2.2.3/intel-2018b-Python-2.7.15/gdal-2.2.3/swig/python'
make[1]: *** [build] Error 2
make[1]: Leaving directory `/tmp/casparl/GDAL/2.2.3/intel-2018b-Python-2.7.15/gdal-2.2.3/swig'
make: *** [swig-modules] Error 2

What we know so far

• Build fails only on our RHEL7 system, builds fine on our Debian9 system.
• Comparing logs of serial builds on the RHEL7 and Debian9 systems shows that on the Debian9 system, the configure step is somewhat different due to the presence of system libraries. However, the step in which the build fails does not seem to be affected by that and is executed on both systems (though the libraries it link could of course be different).
• If I dump an environment script with eb ... --dump-env-script, source that, and run the last icpc command, it fails with the same error. If I dump an environment script with eb ... --dump-env-script, source that, do a make clean and then make -j 24, the build completes succesfully.
• A serial (--parallel=1) on the RHEL7 system also fails.
• @boegel suggested it may have to do with the ulimits set, particularly ulimit -s. However, these were already set the unlimited (confirmed with prebuildopts = "ulimit -a && ").

Workaround

• Adding configopts += ' CFLAGS="$CFLAGS -no-ipo -ip" CXXFLAGS="$CXXFLAGS -no-ipo -ip"' works. This can be used as a workaround, though at the cost of disabling completely the interprocedural optimization (IPO).

Original GDAL-2.2.3-intel-2018b-Python-2.7.15.eb (without workaround)

easyblock = 'ConfigureMake'

name = 'GDAL'
version = '2.2.3'
versionsuffix = '-Python-%(pyver)s'

homepage = 'http://www.gdal.org/'
description = """GDAL is a translator library for raster geospatial data formats that is released under an X/MIT style
 Open Source license by the Open Source Geospatial Foundation. As a library, it presents a single abstract data model
 to the calling application for all supported formats. It also comes with a variety of useful commandline utilities for
 data translation and processing."""

toolchain = {'name': 'intel', 'version': '2018b'}
toolchainopts = {'usempi': True}

source_urls = ['http://download.osgeo.org/gdal/%(version)s/']
sources = [SOURCELOWER_TAR_XZ]

checksums = [
    'a328d63d476b3653f5a25b5f7971e87a15cdf8860ab0729d4b1157ba988b8d0b',  # gdal-2.2.3.tar.xz
]

#prebuildopts = 'ulimit -a && '

dependencies = [
    ('Python', '2.7.15'),
    ('netCDF', '4.6.1'),
    ('expat', '2.2.5'),
    ('GEOS', '3.6.2', '-Python-%(pyver)s'),
    ('SQLite', '3.24.0'),
    ('libxml2', '2.9.8'),
    ('libpng', '1.6.34'),
    ('libjpeg-turbo', '2.0.0'),
    ('JasPer', '2.0.14'),
    ('LibTIFF', '4.0.9'),
    ('zlib', '1.2.11'),
    ('cURL', '7.60.0'),
    ('PCRE', '8.41'),
    ('PROJ', '5.0.0'),
]

configopts = '--with-expat=$EBROOTEXPAT --with-libz=$EBROOTLIBZ --with-hdf5=$EBROOTHDF5 --with-netcdf=$EBROOTNETCDF'
configopts += ' --with-xml2=$EBROOTLIBXML2 --with-geos=$EBROOTGEOS/bin/geos-config --with-jpeg=$EBROOTLIBJPEGMINTURBO'
configopts += ' --with-png=$EBROOTLIBPNG --with-sqlite3=$EBROOTSQLITE --with-jasper=$EBROOTJASPER'
configopts += ' --with-libtiff=$EBROOTLIBTIFF --with-pcre=$EBROOTPCRE --with-python=$EBROOTPYTHON/bin/python'

modextrapaths = {'PYTHONPATH': 'lib/python%(pyshortver)s/site-packages'}

sanity_check_paths = {
    'files': ['lib/libgdal.a', 'lib/libgdal.%s' % SHLIB_EXT],
    'dirs': ['bin', 'include', 'lib/python%(pyshortver)s/site-packages']
}

moduleclass = 'data'

[casparvl]

Just logging this here for future reference, so that people encountering this issue can use the same workaround. I will close the ticket right away, since there is (probably) no action to be taken on the EasyBuild side...

[casparvl]

Reproducing For those who want to reproduce: EasyConfigs for some of the dependencies are not yet in the repo - I just wrote them. Will just add them here for now:

GEOS-3.6.2-intel-2018b-Python-2.7.15.eb

easyblock = 'ConfigureMake'

name = 'GEOS'
version = '3.6.2'
versionsuffix = '-Python-%(pyver)s'

homepage = 'http://trac.osgeo.org/geos'
description = """GEOS (Geometry Engine - Open Source) is a C++ port of the  Java Topology Suite (JTS)"""

toolchain = {'name': 'intel', 'version': '2018b'}

source_urls = ['http://download.osgeo.org/geos/']
sources = [SOURCELOWER_TAR_BZ2]

patches = [
    'GEOS-%(version)s_fix-Python3.patch',
]

checksums = [
    '045a13df84d605a866602f6020fc6cbf8bf4c42fb50de237a08926e1d7d7652a',  # geos-3.6.2.tar.bz2
    'ce320e5533e407807c0e0599b6cf06e207bc993204b27250bf7e1d0f24160029',  # GEOS-3.6.2_fix-Python3.patch
]

dependencies = [('Python', '2.7.15')]

builddependencies = [('SWIG', '3.0.12', versionsuffix)]

configopts = '--enable-python'

modextrapaths = {'PYTHONPATH': 'lib/python%(pyshortver)s/site-packages'}

sanity_check_paths = {
    'files': ['bin/geos-config', 'lib/libgeos.%s' % SHLIB_EXT, 'lib/libgeos.a', 'include/geos.h'],
    'dirs': ['lib/python%(pyshortver)s/site-packages/geos'],
}

moduleclass = 'math'

PROJ-5.0.0-intel-2018b.eb

easyblock = 'ConfigureMake'

name = 'PROJ'
version = '5.0.0'

homepage = 'http://trac.osgeo.org/proj/'
description = """Program proj is a standard Unix filter function which converts
geographic longitude and latitude coordinates into cartesian coordinates"""

toolchain = {'name': 'intel', 'version': '2018b'}
toolchainopts = {'pic': True, 'usempi': True}

source_urls = ['http://download.osgeo.org/proj/']
sources = [SOURCELOWER_TAR_GZ]
checksums = ['b30df08d736e69744cb255828721abb545b494d6032c13a96520f3219a444cd2']

sanity_check_paths = {
    'files': ['bin/cs2cs', 'bin/geod', 'bin/invgeod', 'bin/invproj',
              'bin/nad2bin', 'bin/proj'],
    'dirs': [],
}

moduleclass = 'lib'

[boegel]

Thanks for reporting & documenting this @casparvl!