matplotlib downloads cppy during install - causing build failure when no internet connection is available

[LHurst-JM]

After downloading with eb -Tr --fetch on an internet-connected host, build fails on HPC nodes (on all architectures) without internet access with:

== 2021-08-24 14:30:11,387 build_log.py:169 ERROR EasyBuild crashed with an error (at easybuild/sources/easybuild-framework/easybuild/base/exceptions.py:124 in __init__): cmd " pip install -
-prefix=/mnt/apps/easybuild/2020b-dcore3/EL-7-sky/software/matplotlib/3.3.3-foss-2020b  --no-deps  --ignore-installed  --no-index  --no-build-isolation  ." exited with exit code 1 and output
:
Processing /dev/shm/eb-build/matplotlib/3.3.3/foss-2020b/kiwisolver/kiwisolver-1.3.0
    ERROR: Command errored out with exit status 1:
     command: /mnt/apps/easybuild/2020b/EL-7-sky/software/Python/3.8.6-GCCcore-10.2.0/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/eb-i825n8hh/pip-req-build-zrr81u4k/setup.py'"'"'; __file__='"'"'/tmp/eb-i825n8hh/pip-req-build-zrr81u4k/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/eb-i825n8hh/pip-pip-egg-info-n3503gen
         cwd: /tmp/eb-i825n8hh/pip-req-build-zrr81u4k/
    Complete output (37 lines):
    WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1124)'))': /simple/cppy/
    WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1124)'))': /simple/cppy/
    WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1124)'))': /simple/cppy/
    WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1124)'))': /simple/cppy/
    WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1124)'))': /simple/cppy/
    ERROR: Could not find a version that satisfies the requirement cppy>=1.1.0 (from versions: none)
    ERROR: No matching distribution found for cppy>=1.1.0
    Traceback (most recent call last):
      File "/mnt/apps/easybuild/2020b/EL-7-sky/software/Python/3.8.6-GCCcore-10.2.0/lib/python3.8/site-packages/setuptools/installer.py", line 128, in fetch_build_egg
        subprocess.check_call(cmd)
      File "/mnt/apps/easybuild/2020b/EL-7-sky/software/Python/3.8.6-GCCcore-10.2.0/lib/python3.8/subprocess.py", line 364, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['/mnt/apps/easybuild/2020b/EL-7-sky/software/Python/3.8.6-GCCcore-10.2.0/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/eb-i825n8hh/tmpmmhxlwi5', '--quiet', 'cppy>=1.1.0']' returned non-zero exit status 1.

    The above exception was the direct cause of the following exception:

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/eb-i825n8hh/pip-req-build-zrr81u4k/setup.py", line 59, in <module>
        setup(
      File "/mnt/apps/easybuild/2020b/EL-7-sky/software/Python/3.8.6-GCCcore-10.2.0/lib/python3.8/site-packages/setuptools/__init__.py", line 164, in setup
        _install_setup_requires(attrs)
      File "/mnt/apps/easybuild/2020b/EL-7-sky/software/Python/3.8.6-GCCcore-10.2.0/lib/python3.8/site-packages/setuptools/__init__.py", line 159, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "/mnt/apps/easybuild/2020b/EL-7-sky/software/Python/3.8.6-GCCcore-10.2.0/lib/python3.8/site-packages/setuptools/dist.py", line 699, in fetch_build_eggs
        resolved_dists = pkg_resources.working_set.resolve(
      File "/mnt/apps/easybuild/2020b/EL-7-sky/software/Python/3.8.6-GCCcore-10.2.0/lib/python3.8/site-packages/pkg_resources/__init__.py", line 779, in resolve
        dist = best[req.key] = env.best_match(
      File "/mnt/apps/easybuild/2020b/EL-7-sky/software/Python/3.8.6-GCCcore-10.2.0/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1064, in best_match
        return self.obtain(req, installer)
      File "/mnt/apps/easybuild/2020b/EL-7-sky/software/Python/3.8.6-GCCcore-10.2.0/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1076, in obtain
        return installer(requirement)
      File "/mnt/apps/easybuild/2020b/EL-7-sky/software/Python/3.8.6-GCCcore-10.2.0/lib/python3.8/site-packages/setuptools/dist.py", line 758, in fetch_build_egg
        return fetch_build_egg(self, req)
      File "/mnt/apps/easybuild/2020b/EL-7-sky/software/Python/3.8.6-GCCcore-10.2.0/lib/python3.8/site-packages/setuptools/installer.py", line 130, in fetch_build_egg
        raise DistutilsError(str(e)) from e
    distutils.errors.DistutilsError: Command '['/mnt/apps/easybuild/2020b/EL-7-sky/software/Python/3.8.6-GCCcore-10.2.0/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/eb-i825n8hh/tmpmmhxlwi5', '--quiet', 'cppy>=1.1.0']' returned non-zero exit status 1.
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

I cannot see an existing eb for cppy, I will add one and add it as a dependency to matplotlib.

[LHurst-JM]

With cppy as an external dependency, build then progresses past kiwisolver module but fails with:

running build_ext
    gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -O2 -ftree-vectorize -march=native -fno-math-errno -I/mnt/apps/easybuild/2020b/EL-7-sky/software/FFTW/3.3.8-gompi-2020b/include -I/mnt/apps/easybuild/2020b/EL-7-sky/software/OpenBLAS/0.3.12-GCC-10.2.0/include -fPIC -I/mnt/apps/easybuild/2020b/EL-7-sky/software/Python/3.8.6-GCCcore-10.2.0/include/python3.8 -c /tmp/eb-of383im6/tmpqyi4ptgb.cpp -o tmp/eb-of383im6/tmpqyi4ptgb.o -fvisibility=hidden
    gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -O2 -ftree-vectorize -march=native -fno-math-errno -I/mnt/apps/easybuild/2020b/EL-7-sky/software/FFTW/3.3.8-gompi-2020b/include -I/mnt/apps/easybuild/2020b/EL-7-sky/software/OpenBLAS/0.3.12-GCC-10.2.0/include -fPIC -I/mnt/apps/easybuild/2020b/EL-7-sky/software/Python/3.8.6-GCCcore-10.2.0/include/python3.8 -c /tmp/eb-of383im6/tmpc71b1lmh.cpp -o tmp/eb-of383im6/tmpc71b1lmh.o -fvisibility-inlines-hidden
    gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -O2 -ftree-vectorize -march=native -fno-math-errno -I/mnt/apps/easybuild/2020b/EL-7-sky/software/FFTW/3.3.8-gompi-2020b/include -I/mnt/apps/easybuild/2020b/EL-7-sky/software/OpenBLAS/0.3.12-GCC-10.2.0/include -fPIC -I/mnt/apps/easybuild/2020b/EL-7-sky/software/Python/3.8.6-GCCcore-10.2.0/include/python3.8 -c /tmp/eb-of383im6/tmpao607zxo.cpp -o tmp/eb-of383im6/tmpao607zxo.o -flto
    error: Failed to download FreeType. Please download one of ['https://downloads.sourceforge.net/project/freetype/freetype2/2.6.1/freetype-2.6.1.tar.gz', 'https://download.savannah.gnu.org/releases/freetype/freetype-2.6.1.tar.gz'] and extract it into build/freetype-2.6.1 at the top-level of the source repository.
    ----------------------------------------

[LHurst-JM]

I see the above has already been fixed in the 2021a toolchain eb:

# avoid that matplotlib downloads and builds its own copies of freetype and qhull
_fix_setup = "sed -e 's/#system_freetype = False/system_freetype = True/g' "
_fix_setup += "-e 's/#system_qhull = False/system_qhull = True/g' setup.cfg.template >setup.cfg && "

_include_path = "export CPLUS_INCLUDE_PATH=$EBROOTFREETYPE/include/freetype2:${CPLUS_INCLUDE_PATH} && "

Will backport that fix to 2020b.

[LHurst-JM]

2020b build is failing with, with the backported fix:

building 'matplotlib._qhull' extension
    gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -O2 -ftree-vectorize -march=native -fno-math-errno -fPIC -O2 -ftree-vectorize -march=native -fno-math-errno -I/mnt/apps/easybuild/2020b/EL-7-sky/software/FFTW/3.3.8-gompi-2020b/include -I/mnt/apps/easybuild/2020b/EL-7-sky/software/OpenBLAS/0.3.12-GCC-10.2.0/include -fPIC -DMPL_DEVNULL=/dev/null -DPY_ARRAY_UNIQUE_SYMBOL=MPL_matplotlib__qhull_ARRAY_API -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -D__STDC_FORMAT_MACROS=1 -I/mnt/apps/easybuild/2020b/EL-7-sky/software/SciPy-bundle/2020.11-foss-2020b/lib/python3.8/site-packages/numpy/core/include -I/mnt/apps/easybuild/2020b/EL-7-sky/software/Python/3.8.6-GCCcore-10.2.0/include/python3.8 -c src/qhull_wrap.c -o build/temp.linux-x86_64-3.8/src/qhull_wrap.o -fvisibility=hidden -flto
    gcc -shared -L/mnt/apps/easybuild/2020b/EL-7-sky/software/libffi/3.3-GCCcore-10.2.0/lib64 -L/mnt/apps/easybuild/2020b/EL-7-sky/software/libffi/3.3-GCCcore-10.2.0/lib -L/mnt/apps/easybuild/2020b/EL-7-sky/software/GMP/6.2.0-GCCcore-10.2.0/lib64 -L/mnt/apps/easybuild/2020b/EL-7-sky/software/GMP/6.2.0-GCCcore-10.2.0/lib -L/mnt/apps/easybuild/2020b/EL-7-sky/software/XZ/5.2.5-GCCcore-10.2.0/lib64 -L/mnt/apps/easybuild/2020b/EL-7-sky/software/XZ/5.2.5-GCCcore-10.2.0/lib -L/mnt/apps/easybuild/2020b/EL-7-sky/software/SQLite/3.33.0-GCCcore-10.2.0/lib64 -L/mnt/apps/easybuild/2020b/EL-7-sky/software/SQLite/3.33.0-GCCcore-10.2.0/lib -L/mnt/apps/easybuild/2020b/EL-7-sky/software/ncurses/6.2-GCCcore-10.2.0/lib64 -L/mnt/apps/easybuild/2020b/EL-7-sky/software/ncurses/6.2-GCCcore-10.2.0/lib -L/mnt/apps/easybuild/2020b/EL-7-sky/software/libreadline/8.0-GCCcore-10.2.0/lib64 -L/mnt/apps/easybuild/2020b/EL-7-sky/software/libreadline/8.0-GCCcore-10.2.0/lib -L/mnt/apps/easybuild/2020b/EL-7-sky/software/zlib/1.2.11-GCCcore-10.2.0/lib64 -L/mnt/apps/easybuild/2020b/EL-7-sky/software/zlib/1.2.11-GCCcore-10.2.0/lib -L/mnt/apps/easybuild/2020b/EL-7-sky/software/bzip2/1.0.8-GCCcore-10.2.0/lib64 -L/mnt/apps/easybuild/2020b/EL-7-sky/software/bzip2/1.0.8-GCCcore-10.2.0/lib -L/mnt/apps/easybuild/2020b/EL-7-sky/software/binutils/2.35-GCCcore-10.2.0/lib64 -L/mnt/apps/easybuild/2020b/EL-7-sky/software/binutils/2.35-GCCcore-10.2.0/lib -L/mnt/apps/easybuild/2020b/EL-7-sky/software/GCCcore/10.2.0/lib64 -L/mnt/apps/easybuild/2020b/EL-7-sky/software/GCCcore/10.2.0/lib -L/mnt/apps/easybuild/2020b/EL-7-sky/software/libffi/3.3-GCCcore-10.2.0/lib64 -L/mnt/apps/easybuild/2020b/EL-7-sky/software/libffi/3.3-GCCcore-10.2.0/lib -L/mnt/apps/easybuild/2020b/EL-7-sky/software/GMP/6.2.0-GCCcore-10.2.0/lib64 -L/mnt/apps/easybuild/2020b/EL-7-sky/software/GMP/6.2.0-GCCcore-10.2.0/lib -L/mnt/apps/easybuild/2020b/EL-7-sky/software/XZ/5.2.5-GCCcore-10.2.0/lib64 -L/mnt/apps/easybuild/2020b/EL-7-sky/software/XZ/5.2.5-GCCcore-10.2.0/lib -L/mnt/apps/easybuild/2020b/EL-7-sky/software/SQLite/3.33.0-GCCcore-10.2.0/lib64 -L/mnt/apps/easybuild/2020b/EL-7-sky/software/SQLite/3.33.0-GCCcore-10.2.0/lib -L/mnt/apps/easybuild/2020b/EL-7-sky/software/ncurses/6.2-GCCcore-10.2.0/lib64 -L/mnt/apps/easybuild/2020b/EL-7-sky/software/ncurses/6.2-GCCcore-10.2.0/lib -L/mnt/apps/easybuild/2020b/EL-7-sky/software/libreadline/8.0-GCCcore-10.2.0/lib64 -L/mnt/apps/easybuild/2020b/EL-7-sky/software/libreadline/8.0-GCCcore-10.2.0/lib -L/mnt/apps/easybuild/2020b/EL-7-sky/software/zlib/1.2.11-GCCcore-10.2.0/lib64 -L/mnt/apps/easybuild/2020b/EL-7-sky/software/zlib/1.2.11-GCCcore-10.2.0/lib -L/mnt/apps/easybuild/2020b/EL-7-sky/software/bzip2/1.0.8-GCCcore-10.2.0/lib64 -L/mnt/apps/easybuild/2020b/EL-7-sky/software/bzip2/1.0.8-GCCcore-10.2.0/lib -L/mnt/apps/easybuild/2020b/EL-7-sky/software/binutils/2.35-GCCcore-10.2.0/lib64 -L/mnt/apps/easybuild/2020b/EL-7-sky/software/binutils/2.35-GCCcore-10.2.0/lib -L/mnt/apps/easybuild/2020b/EL-7-sky/software/GCCcore/10.2.0/lib64 -L/mnt/apps/easybuild/2020b/EL-7-sky/software/GCCcore/10.2.0/lib -L/mnt/apps/easybuild/2020b/EL-7-sky/software/FFTW/3.3.8-gompi-2020b/lib64 -L/mnt/apps/easybuild/2020b/EL-7-sky/software/FFTW/3.3.8-gompi-2020b/lib -L/mnt/apps/easybuild/2020b/EL-7-sky/software/ScaLAPACK/2.1.0-gompi-2020b/lib64 -L/mnt/apps/easybuild/2020b/EL-7-sky/software/ScaLAPACK/2.1.0-gompi-2020b/lib -L/mnt/apps/easybuild/2020b/EL-7-sky/software/OpenBLAS/0.3.12-GCC-10.2.0/lib64 -L/mnt/apps/easybuild/2020b/EL-7-sky/software/OpenBLAS/0.3.12-GCC-10.2.0/lib -L/mnt/apps/easybuild/2020b/EL-7-sky/software/GCCcore/10.2.0/lib64 -L/mnt/apps/easybuild/2020b/EL-7-sky/software/GCCcore/10.2.0/lib -O2 -ftree-vectorize -march=native -fno-math-errno -I/mnt/apps/easybuild/2020b/EL-7-sky/software/FFTW/3.3.8-gompi-2020b/include -I/mnt/apps/easybuild/2020b/EL-7-sky/software/OpenBLAS/0.3.12-GCC-10.2.0/include build/temp.linux-x86_64-3.8/src/qhull_wrap.o -L/mnt/apps/easybuild/2020b/EL-7-sky/software/Python/3.8.6-GCCcore-10.2.0/lib -lqhull -o build/lib.linux-x86_64-3.8/matplotlib/_qhull.cpython-38-x86_64-linux-gnu.so
    /mnt/apps/easybuild/2020b/EL-7-sky/software/binutils/2.35-GCCcore-10.2.0/bin/ld.gold: error: cannot find -lqhull
    collect2: error: ld returned 1 exit status
    error: command 'gcc' failed with exit status 1
    ----------------------------------------

2021a build passes, apart from the dependecy version numbers the Qhull eb file is identical:

$ diff Qhull-2020.2-GCCcore-10.{2,3}.0.eb
16c16
< toolchain = {'name': 'GCCcore', 'version': '10.2.0'}
---
> toolchain = {'name': 'GCCcore', 'version': '10.3.0'}
24,25c24,25
<     ('binutils', '2.35'),
<     ('CMake', '3.18.4'),
---
>     ('binutils', '2.36.1'),
>     ('CMake', '3.20.1'),

And matplotlib's diff a bit more convoluted but I can't see an obvious cause:

4c4
< version = '3.3.3'
---
> version = '3.4.2'
11c11
< toolchain = {'name': 'foss', 'version': '2020b'}
---
> toolchain = {'name': 'foss', 'version': '2021a'}
18,19c18,19
<     ('Python', '3.8.6'),
<     ('SciPy-bundle', '2020.11'),
---
>     ('Python', '3.9.5'),
>     ('SciPy-bundle', '2021.05'),
21c21
<     ('freetype', '2.10.3'),
---
>     ('freetype', '2.10.4'),
23c23
<     ('Pillow', '8.0.1'),
---
>     ('Pillow', '8.2.0'),
40a41
>         'source_urls': ['https://pypi.python.org/packages/source/C/Cycler'],
43,44c44,46
<     ('kiwisolver', '1.3.0', {
<         'checksums': ['14f81644e1f3bf01fbc8b9c990a7889e9bb4400c4d0ff9155aa0bdd19cce24a9'],
---
>     ('kiwisolver', '1.3.1', {
>         'source_urls': ['https://pypi.python.org/packages/source/k/kiwisolver'],
>         'checksums': ['950a199911a8d94683a6b10321f9345d5a3a8433ec58b217ace979e18f16e248'],
48c50,51
<         'checksums': ['b1b60c6476c4cfe9e5cf8ab0d3127476fd3d5f05de0f343a452badaad0e4bdec'],
---
>         'source_urls': ['https://pypi.python.org/packages/source/m/matplotlib'],
>         'checksums': ['d8d994cefdff9aaba45166eb3de4f5211adb4accac85cbf97137e98f26ea0219'],

My plan is to try the 3.4.2 matplotlib in my 2020b environment, as I suspect the library issue may be related to the version - looking at matplotlibs website sources the dependencies page is entirely new in 3.4.x so I suspect they did a lot of work here between the releases.

The fix for this issue (cppy dependency) is working with the 2021a toolchain, so I will submit that as a PR shortly.

[LHurst-JM]

3.4.2 built fine in the 2020b toolchain - the external qhull problem is clearly version related.

I think the best thing for me to do is to submit the fix for the 2021a toolchain and I'll keep the extra version (3.4.2-foss-2020b) local to our environment.