search

easybuilders / easybuild-easyblocks

Collection of easyblocks that implement support for building and installing software with EasyBuild.
https://easybuild.io
GNU General Public License v2.0
106 stars 285 forks source link

allow Python bundles and packages to specify a maximum Python version for the system toolchain #3431

Closed casparvl closed 1 month ago

casparvl commented 2 months ago

We implement support for specifying a maximum python version for EasyConfigs that use system python (through the max_py_majver, max_py_minver keywords). This is is required for https://github.com/easybuilders/easybuild-easyconfigs/pull/21307 which has one EasyConfig that will only work on the python 3.6 - 3.11 range (3.12 will fail because the setuptools is too old, but newer setuptools do not support python3.6).

This PR is built on top of https://github.com/easybuilders/easybuild-easyblocks/pull/3430 . That one should probably be reviewed & merged first, after which this PR should be synced with develop.

bedroge commented 2 months ago

Test report by @bedroge

Overview of tested easyconfigs (in order)

Build succeeded for 4 out of 4 (4 easyconfigs in total) bob-Latitude-5300 - Linux Ubuntu 24.04.1 LTS (Noble Numbat), x86_64, Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz, Python 3.12.3 See https://gist.github.com/bedroge/3235d0ff53051a7f7472f287c85e5194 for a full test report.

bedroge commented 2 months ago

Test report by @bedroge

Overview of tested easyconfigs (in order)

Build succeeded for 0 out of 1 (1 easyconfigs in total) bob-Latitude-5300 - Linux Ubuntu 24.04.1 LTS (Noble Numbat), x86_64, Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz, Python 3.12.3 See https://gist.github.com/bedroge/c5bc91e9bf2dab40ee1d2eec64f6eba2 for a full test report.

Expected failure that confirms that the new functionality works. I used Python 3.12 for a ReFrame version (from https://github.com/easybuilders/easybuild-easyconfigs/pull/21307) that only supports Python <= 3.11.

== FAILED: Installation ended unsuccessfully (build directory: /data/eb/build/ReFrame/4.6.2/system-system-Python-3.6): build failed (first 300 chars): Failed to pick Python command that satisfies requirements in the easyconfig (req_py_majver = 3, req_py_minver = 6, max_py_majver = 3, max_py_minver = 11) (took 0 secs)
bedroge commented 2 months ago

Test report by @bedroge

Overview of tested easyconfigs (in order)

Build succeeded for 1 out of 1 (1 easyconfigs in total) bob-Latitude-5300 - Linux Ubuntu 24.04.1 LTS (Noble Numbat), x86_64, Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz, Python 3.12.3 See https://gist.github.com/bedroge/0a9eb4ce1c0a45c11a91de2ef7e08122 for a full test report.

bedroge commented 2 months ago

Test report by @bedroge

Overview of tested easyconfigs (in order)

Build succeeded for 0 out of 1 (1 easyconfigs in total) bob-Latitude-5300 - Linux Ubuntu 24.04.1 LTS (Noble Numbat), x86_64, Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz, Python 3.12.3 See https://gist.github.com/bedroge/5459719d39a8ec3d367525b0a0eb14cc for a full test report.

bedroge commented 2 months ago

Redoing all test builds to make sure things still work.

bedroge commented 2 months ago

Test report by @bedroge

Overview of tested easyconfigs (in order)

Build succeeded for 5 out of 5 (5 easyconfigs in total) bob-Latitude-5300 - Linux Ubuntu 24.04.1 LTS (Noble Numbat), x86_64, Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz, Python 3.12.3 See https://gist.github.com/bedroge/9e6d8b842a542e42333cb212c2c2b18b for a full test report.

bedroge commented 2 months ago

Test report by @bedroge

Overview of tested easyconfigs (in order)

Build succeeded for 0 out of 1 (1 easyconfigs in total) bob-Latitude-5300 - Linux Ubuntu 24.04.1 LTS (Noble Numbat), x86_64, Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz, Python 3.12.3 See https://gist.github.com/bedroge/62e41ef10b452bb177eeb4c9d67484c7 for a full test report.

Reworded the debug message a bit:

Python version (3.12.3) on the system is newer than the maximum supported Python version specified in the easyconfig (3.11)
bedroge commented 2 months ago

Test report by @bedroge

Overview of tested easyconfigs (in order)

Build succeeded for 1 out of 1 (1 easyconfigs in total) bob-Latitude-5300 - Linux Ubuntu 24.04.1 LTS (Noble Numbat), x86_64, Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz, Python 3.12.3 See https://gist.github.com/bedroge/1a18889f623416471276207030f5d75f for a full test report.

bedroge commented 2 months ago

@casparvl I did a search for pick_python_cmd, and found that there are unit tests for this function. So it would be good to add some additional ones here https://github.com/easybuilders/easybuild-easyblocks/blob/develop/test/easyblocks/module.py#L227.

edit: added two tests in https://github.com/casparvl/easybuild-easyblocks/pull/4.

casparvl commented 2 months ago

@bedroge I merged your unit test. Is there anything else that needs to be done for this PR to be merged? :)

bedroge commented 1 month ago

Test report by @bedroge

Overview of tested easyconfigs (in order)

Build succeeded for 5 out of 5 (5 easyconfigs in total) bob-Latitude-5300 - Linux Ubuntu 24.04.1 LTS (Noble Numbat), x86_64, Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz, Python 3.12.3 See https://gist.github.com/bedroge/e084b7ecccf94c2730b4362c9e8887d3 for a full test report.

Flamefire commented 1 month ago

@casparvl What is the intended semantic for specifying only a major version? I would expect that max_py_majver = 3 would accept any Python 3 version. Or maybe more sensible: max_py_majver = 2 would only exclude Python 3. Currently the implementation seems to check for < max_py_majver, which means Python 3.1 and Python 2.1 respectively would be excluded.