search

easybuilders / easybuild-easyblocks

Collection of easyblocks that implement support for building and installing software with EasyBuild.
https://easybuild.io
GNU General Public License v2.0
104 stars 284 forks source link

Regression: Broken templating in EasyBuild 4.7.0 #2881

Closed mboisson closed 1 year ago

mboisson commented 1 year ago

I am not sure exactly when it happened, but recent EasyBuild versions broke template replacement in configopts when using multi_deps, i.e. : 

multi_deps = {'Python': ['3.10', '3.9', '3.8'] }
configopts = "-DPYQT5_SIP_DIR=$EBROOTQT5/share/python%(pyshortver)s/site-packages/sip/PyQt5 -DPYQT5_MOD_DIR=$EBROOTQT5/lib/python%(pyshortver)s/site-packages/PyQt5 "

for a CMakeMake EasyBlock (maybe others, I have not tested) no longer yield builds with different configopts argument based on %(pyshortver)s.

In a previous build log, %(pyshortver)s gets replaced for every iteration: 

grep run.py /cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/gcc9/qgis/3.16.10/easybuild/*.log | grep cmake | grep running
== 2021-08-31 19:14:35,595 run.py:233 INFO running cmd:  cmake -DCMAKE_INSTALL_PREFIX=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/gcc9/qgis/3.16.10 -DCMAKE_BUILD_TYPE=Release -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_C_COMPILER='gcc' -DCMAKE_C_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno -fPIC -std=gnu++11' -DCMAKE_CXX_COMPILER='g++' -DCMAKE_CXX_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno -fPIC -std=gnu++11' -DCMAKE_Fortran_COMPILER='gfortran' -DCMAKE_Fortran_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno -fPIC' -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_FIND_USE_PACKAGE_REGISTRY=FALSE  -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON  -DENABLE_XHOST=OFF  -DCMAKE_SKIP_INSTALL_RPATH=ON -DCMAKE_PREFIX_PATH=$EBROOTQT -DQJSON_DIR=$EBROOTQJSON/lib/cmake/qjson -DQSCINTILLA_LIBRARY=$EBROOTQSCINTILLA/lib/libqscintilla2_qt5.so -DWITH_QTWEBKIT=OFF -DPYQT5_SIP_DIR=$EBROOTQT5/share/python3.7/sip/PyQt5 -DQSCI_SIP_DIR=$EBROOTQSCINTILLA/share/sip/PyQt5 /tmp/ebuser/avx2/QGIS/3.16.10/GCC-9.3.0/qgis-3.16.10/
== 2021-08-31 22:14:21,025 run.py:233 INFO running cmd:  cmake -DCMAKE_INSTALL_PREFIX=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/gcc9/qgis/3.16.10 -DCMAKE_BUILD_TYPE=Release -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_C_COMPILER='gcc' -DCMAKE_C_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno -fPIC -std=gnu++11' -DCMAKE_CXX_COMPILER='g++' -DCMAKE_CXX_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno -fPIC -std=gnu++11' -DCMAKE_Fortran_COMPILER='gfortran' -DCMAKE_Fortran_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno -fPIC' -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_FIND_USE_PACKAGE_REGISTRY=FALSE  -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON  -DENABLE_XHOST=OFF  -DCMAKE_SKIP_INSTALL_RPATH=ON -DCMAKE_PREFIX_PATH=$EBROOTQT -DQJSON_DIR=$EBROOTQJSON/lib/cmake/qjson -DQSCINTILLA_LIBRARY=$EBROOTQSCINTILLA/lib/libqscintilla2_qt5.so -DWITH_QTWEBKIT=OFF -DPYQT5_SIP_DIR=$EBROOTQT5/share/python3.8/sip/PyQt5 -DQSCI_SIP_DIR=$EBROOTQSCINTILLA/share/sip/PyQt5 /tmp/ebuser/avx2/QGIS/3.16.10/GCC-9.3.0/qgis-3.16.10/
== 2021-08-31 23:54:14,144 run.py:233 INFO running cmd:  cmake -DCMAKE_INSTALL_PREFIX=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/gcc9/qgis/3.16.10 -DCMAKE_BUILD_TYPE=Release -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_C_COMPILER='gcc' -DCMAKE_C_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno -fPIC -std=gnu++11' -DCMAKE_CXX_COMPILER='g++' -DCMAKE_CXX_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno -fPIC -std=gnu++11' -DCMAKE_Fortran_COMPILER='gfortran' -DCMAKE_Fortran_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno -fPIC' -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_FIND_USE_PACKAGE_REGISTRY=FALSE  -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON  -DENABLE_XHOST=OFF  -DCMAKE_SKIP_INSTALL_RPATH=ON -DCMAKE_PREFIX_PATH=$EBROOTQT -DQJSON_DIR=$EBROOTQJSON/lib/cmake/qjson -DQSCINTILLA_LIBRARY=$EBROOTQSCINTILLA/lib/libqscintilla2_qt5.so -DWITH_QTWEBKIT=OFF -DPYQT5_SIP_DIR=$EBROOTQT5/share/python3.9/sip/PyQt5 -DQSCI_SIP_DIR=$EBROOTQSCINTILLA/share/sip/PyQt5 /tmp/ebuser/avx2/QGIS/3.16.10/GCC-9.3.0/qgis-3.16.10/

while in a recent build, it keeps the same value as for the first iteration: 

$ grep run.py /tmp/eb-3yw8sr68/easybuild-6ry1d4vr.log | grep cmake | grep running
== 2023-02-08 16:18:35,282 run.py:236 INFO running cmd: patch -p1 -i /home/mboisson/git/easybuild-easyconfigs/easybuild/easyconfigs/q/QGIS/QGIS-3.22.14-FindPyQt5.cmake.patch
== 2023-02-08 16:18:56,589 run.py:236 INFO running cmd:  cmake  -DCMAKE_INSTALL_PREFIX=/home/mboisson/.local/easybuild/software/2020/avx2/Compiler/gcc9/qgis/3.22.14 -DCMAKE_BUILD_TYPE=Release -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF  -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON  -DENABLE_XHOST=OFF  -DCMAKE_SKIP_INSTALL_RPATH=ON -DWITH_QTWEBKIT=OFF -DPYQT5_SIP_DIR=$EBROOTQT5/share/python3.10/site-packages/sip/PyQt5 -DPYQT5_MOD_DIR=$EBROOTQT5/lib/python3.10/site-packages/PyQt5 -DQSCI_SIP_DIR=$EBROOTQSCINTILLA/share/sip/PyQt5 -DPYQT5_VERSION_STR=5.15.5  /tmp/mboisson/avx2/QGIS/3.22.14/GCC-9.3.0/qgis-3.22.14/
== 2023-02-08 16:31:47,766 run.py:236 INFO running cmd: patch -p1 -i /home/mboisson/git/easybuild-easyconfigs/easybuild/easyconfigs/q/QGIS/QGIS-3.22.14-FindPyQt5.cmake.patch
== 2023-02-08 16:32:07,346 run.py:236 INFO running cmd:  cmake   -DCMAKE_INSTALL_PREFIX=/home/mboisson/.local/easybuild/software/2020/avx2/Compiler/gcc9/qgis/3.22.14 -DCMAKE_BUILD_TYPE=Release -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF  -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON  -DENABLE_XHOST=OFF  -DCMAKE_SKIP_INSTALL_RPATH=ON -DWITH_QTWEBKIT=OFF -DPYQT5_SIP_DIR=$EBROOTQT5/share/python3.10/site-packages/sip/PyQt5 -DPYQT5_MOD_DIR=$EBROOTQT5/lib/python3.10/site-packages/PyQt5 -DQSCI_SIP_DIR=$EBROOTQSCINTILLA/share/sip/PyQt5 -DPYQT5_VERSION_STR=5.15.5  /tmp/mboisson/avx2/QGIS/3.22.14/GCC-9.3.0/qgis-3.22.14/
mboisson commented 1 year ago

According to this, it worked at least up to version 4.6.0: 

for f in $(grep -rl "configopts.*pyshortver" /cvmfs/soft.computecanada.ca/easybuild/ebfiles_repo/2020/); do grep easybuild-framework_version $f; done
    "easybuild-framework_version": "4.4.1-r73a32642a48430484a7c9a4654e7afda27268be4",
    "easybuild-framework_version": "4.3.3-r15719a0ece2011cea9ce176d1763cdc620b188f4",
    "easybuild-framework_version": "4.6.0-ra6dbb4b22832c431590f725219d89d82c9946dba",
    "easybuild-framework_version": "4.6.0-ra6dbb4b22832c431590f725219d89d82c9946dba",
    "easybuild-framework_version": "4.6.0-ra6dbb4b22832c431590f725219d89d82c9946dba",
    "easybuild-framework_version": "4.6.0-ra6dbb4b22832c431590f725219d89d82c9946dba",
    "easybuild-framework_version": "4.3.3-r3ff498d1f735c51b313c04c5e4abeeaecc23589b",
    "easybuild-framework_version": "4.3.3-r3ff498d1f735c51b313c04c5e4abeeaecc23589b",
    "easybuild-framework_version": "4.2.2-r64415f53d9bdff498788ce494f68a98641a3883d",
    "easybuild-framework_version": "4.2.2-r64415f53d9bdff498788ce494f68a98641a3883d",
    "easybuild-framework_version": "4.4.0-r5824ede87a67bf66b912bcb3f7d73c27835716af",
    "easybuild-framework_version": "4.4.0-r5824ede87a67bf66b912bcb3f7d73c27835716af",
    "easybuild-framework_version": "4.4.0-r5824ede87a67bf66b912bcb3f7d73c27835716af",
    "easybuild-framework_version": "4.3.3-r15719a0ece2011cea9ce176d1763cdc620b188f4",

Looking at one of these recipes, built with EasyBuild 4.6.0: 

grep "run.py" /cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/MPI/gcc9/openmpi4/hoomd-blue-mpi/3.6.0/easybuild/easybuild-HOOMD-blue-3.6.0-20221117.183917.log | grep cmake | grep running
== 2022-11-17 18:03:15,250 run.py:233 INFO running cmd:  cmake -DCMAKE_INSTALL_PREFIX=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/MPI/gcc9/openmpi4/hoomd-blue-mpi/3.6.0 -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER='mpicc' -DCMAKE_C_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno' -DCMAKE_CXX_COMPILER='mpicxx' -DCMAKE_CXX_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno' -DCMAKE_Fortran_COMPILER='mpifort' -DCMAKE_Fortran_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno' -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_FIND_USE_PACKAGE_REGISTRY=FALSE  -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON  -DENABLE_XHOST=OFF  -DCMAKE_SKIP_INSTALL_RPATH=ON -DENABLE_MPI=ON -DENABLE_CUDA=OFF -DENABLE_TBB=ON -DENABLE_LLVM=OFF -DCMAKE_INSTALL_PREFIX=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/MPI/gcc9/openmpi4/hoomd-blue-mpi/3.6.0/lib/python3.8/site-packages  /tmp/ebuser/avx2/HOOMDblue/3.6.0/gompi-2020a-mpi/hoomd-v3.6.0/
== 2022-11-17 18:15:51,643 run.py:233 INFO running cmd:  cmake -DCMAKE_INSTALL_PREFIX=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/MPI/gcc9/openmpi4/hoomd-blue-mpi/3.6.0 -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER='mpicc' -DCMAKE_C_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno' -DCMAKE_CXX_COMPILER='mpicxx' -DCMAKE_CXX_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno' -DCMAKE_Fortran_COMPILER='mpifort' -DCMAKE_Fortran_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno' -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_FIND_USE_PACKAGE_REGISTRY=FALSE  -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON  -DENABLE_XHOST=OFF  -DCMAKE_SKIP_INSTALL_RPATH=ON -DENABLE_MPI=ON -DENABLE_CUDA=OFF -DENABLE_TBB=ON -DENABLE_LLVM=OFF -DCMAKE_INSTALL_PREFIX=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/MPI/gcc9/openmpi4/hoomd-blue-mpi/3.6.0/lib/python3.9/site-packages  /tmp/ebuser/avx2/HOOMDblue/3.6.0/gompi-2020a-mpi/hoomd-v3.6.0/
== 2022-11-17 18:27:07,729 run.py:233 INFO running cmd:  cmake -DCMAKE_INSTALL_PREFIX=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/MPI/gcc9/openmpi4/hoomd-blue-mpi/3.6.0 -DCMAKE_BUILD_TYPE=Release -DCMAKE_C_COMPILER='mpicc' -DCMAKE_C_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno' -DCMAKE_CXX_COMPILER='mpicxx' -DCMAKE_CXX_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno' -DCMAKE_Fortran_COMPILER='mpifort' -DCMAKE_Fortran_FLAGS='-O2 -ftree-vectorize -march=core-avx2 -fno-math-errno' -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_FIND_USE_PACKAGE_REGISTRY=FALSE  -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON  -DENABLE_XHOST=OFF  -DCMAKE_SKIP_INSTALL_RPATH=ON -DENABLE_MPI=ON -DENABLE_CUDA=OFF -DENABLE_TBB=ON -DENABLE_LLVM=OFF -DCMAKE_INSTALL_PREFIX=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/MPI/gcc9/openmpi4/hoomd-blue-mpi/3.6.0/lib/python3.10/site-packages  /tmp/ebuser/avx2/HOOMDblue/3.6.0/gompi-2020a-mpi/hoomd-v3.6.0/
mboisson commented 1 year ago

This commit from @Flamefire is my first suspect. It messes with configopts and has no explicit code to handle templates

https://github.com/easybuilders/easybuild-easyblocks/commit/0b78b0280ebbd8978110ce3853e265e515425a40

branfosj commented 1 year ago

If I've got my testing correct:

mboisson commented 1 year ago

If I've got my testing correct:

  • This broke between 4.6.0 and 4.6.1
  • This impacts CMake only - not ConfigureMake

That would match the above commit, I think

boegel commented 1 year ago

I confirmed that this can not be reproduced using MEGAHIT-1.2.9-GCCcore-9.3.0.eb (which uses multi_deps + CMakeMake) when fiddling with preconfigopts, so looks specific to CMakeMake and configopts.

This mean is indeed a likely culprit, so I'll move this issue into the easyblocks repo...

mboisson commented 1 year ago

There's another recently opened issue (https://github.com/easybuilders/easybuild-easyconfigs/issues/17259) caused by this PR https://github.com/easybuilders/easybuild-easyblocks/pull/2514

mboisson commented 1 year ago

To avoid running into this in the future, it may be worth displaying a warning when __setitem__ is called with enable_templating = True in easyconfig.py ?

Flamefire commented 1 year ago

To avoid running into this in the future, it may be worth displaying a warning when __setitem__ is called with enable_templating = True in easyconfig.py ?

Actually the issue is __getitem__ as __setitem__ doesn't do any templating. Furthermore we do a lot of modifications with enabled templating without running into this issue.

I haven't found any of our ECs to be affected by this. If anyone can share one to test this that would help. Cmd used: grep -F configopts $(grep -F --files-with-matches multi_deps $(grep -rF --files-with-matches CMakeMake easybuild-easyconfigs/easybuild/easyconfigs)) | grep -F '%(py'

Flamefire commented 1 year ago

That PR may have another regression related to iterating options: First time the configopts are set/changed they are as specified by the easyconfig. During the next iteration they are already changed.

Example (after fix) of a single run with 2 python versions:

configopts=-DPYVER=%(pyshortver)s
  running command "cmake  -DCMAKE_INSTALL_PREFIX=/beegfs/ws/1/s3248973-EasyBuild/easybuild-haswell/software/MetaBAT/2.14-gompi-2019a -DBoost_NO_SYSTEM_PATHS=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF -DBOOST_ROOT=/sw/installed/Boost/1.70.0-gompi-2019a -DPYVER=3.7 /tmp/s3248973-EasyBuild/MetaBAT/2.14/gompi-2019a/MetaBAT-2.14/"

configopts=-DCMAKE_INSTALL_PREFIX=/tmp/easybuild-tmp/eb-D18oYP/__ROOT__/beegfs/ws/1/s3248973-EasyBuild/easybuild-haswell/software/MetaBAT/2.14-gompi-2019a -DBoost_NO_SYSTEM_PATHS=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF -DBOOST_ROOT=/sw/installed/Boost/1.70.0-gompi-2019a -DPYVER=%(pyshortver)s
  running command "cmake   -DCMAKE_INSTALL_PREFIX=/beegfs/ws/1/s3248973-EasyBuild/easybuild-haswell/software/MetaBAT/2.14-gompi-2019a -DBoost_NO_SYSTEM_PATHS=ON -DCMAKE_BUILD_TYPE=Release -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF -DBOOST_ROOT=/sw/installed/Boost/1.70.0-gompi-2019a -DPYVER=2.7 /tmp/s3248973-EasyBuild/MetaBAT/2.14/gompi-2019a/MetaBAT-2.14/"
mboisson commented 1 year ago

To avoid running into this in the future, it may be worth displaying a warning when __setitem__ is called with enable_templating = True in easyconfig.py ?

Actually the issue is __getitem__ as __setitem__ doesn't do any templating. Furthermore we do a lot of modifications with enabled templating without running into this issue.

I disagree the issue is __getitem__. While __getitem__ is what resolves the templates, __setitem__ is what wrongly replaces values with templates unresolved by values with templates resolved. My point is that one should probably never use __setitem__ with template values resolved.

It is reasonable to call __getitem__ without disabling templating (that's what it's for, obviously), but it is unreasonable to call __setitem__ with the value obtained from __getitem__ with templates resolved. The dangerous action here is __setitem__

I haven't found any of our ECs to be affected by this. If anyone can share one to test this that would help. Cmd used: grep -F configopts $(grep -F --files-with-matches multi_deps $(grep -rF --files-with-matches CMakeMake easybuild-easyconfigs/easybuild/easyconfigs)) | grep -F '%(py'

You will find many in our fork https://github.com/ComputeCanada/easybuild-easyconfigs

mboisson commented 1 year ago

At least, I would add a test like the one proposed in https://github.com/easybuilders/easybuild-framework/pull/4212

so that the build fails hard and clearly, rather than failing silently and letting the user to ponder

Flamefire commented 1 year ago

There's another recently opened issue (easybuilders/easybuild-easyconfigs#17259) caused by this PR #2514

That is actually a different issue. I fixed that with a patch in https://github.com/easybuilders/easybuild-easyconfigs/pull/17288

At least, I would add a test like the one proposed in easybuilders/easybuild-framework#4212

so that the build fails hard and clearly, rather than failing silently and letting the user to ponder

Not sure how we can reasonably do that as we do self.cfg[...] = ... or even self.cfg[...] += ... almost everywhere. And even self.cfg.update (which we also use a lot) doesn't handle the templating although there are drawbacks doing that (deduplication may fail).