run `pip check` only once for PythonBundle

Flamefire commented 2 months ago

(created using eb --new-pr)

We have 2 checks in PythonPackage:

pip check
pip list -> Check for "0.0.0" versions

In PythonBundle those are run for every extension after the build of the whole EC even though running it once is enough because the result will always be the same.

This PR uses the following logic:

sanity_pip_check should be set at the top of PythonBundle and not for the individual extensions. Currently if any extension has it enabled the check will be run so it does not make sense to disable/enable it for individual extensions. PythonBundle passes its value for this to every extension as a default so a deprecation is added in case it gets changed in an extension.

Similar reasoning applies to unversioned_packages: Only a single value for the whole bundle is useful and hence should be set at the top. For kind of backwards compatibility during the deprecation an union of all those values is used in the check.

PythonPackage does no longer do the pip checks if it is an extension and the parent EC (e.g. PythonBundle) has a value for sanity_pip_check set.

PythonBundle does the pip check if itself or any extension has requested it issuing a deprecation if something differs.

Refactoring

To make this possible some refactoring was required.
This makes the diff look large although it is mostly moved code. Explanation follows to help navigate the changes

run_pip_check is moved out of sanity_check_step of PythonPackage such that it can be used by PythonBundle
This required moving the dependent method det_installed_python_packages out of the class too, the original PythonPackage.get_installed_python_packages needs to stay for backwards compatibility which prevents giving the same name to the free function. Maybe in EB 5 we can remove it and use get_installed_python_packages for the global method? det_-prefix is chosen similar to det_py_libdirs
PythonBundle.sanity_check_step now requires python_cmd to be available which was only set in the prepare_step that is skipped in --sanity-check-only --> Factor out prepare_python from prepare_step similar to PythonPackage
There was a mismatch in the code to detect the python command to use although I see no reason for that. I factored out find_python_cmd from PythonPackage.prepare_python and call it from PythonBundle. I left the check for a loaded Python module in PythonBundle as I don't know the reason for that check. IMO it should either be in both or neither

Fixes #3418

Flamefire commented 2 months ago

Test report by @Flamefire

Overview of tested easyconfigs (in order)

SUCCESS SciPy-bundle-2022.05-foss-2022a.eb
SUCCESS SciPy-bundle-2023.02-gfbf-2022b.eb
SUCCESS Python-bundle-PyPI-2023.06-GCCcore-12.3.0.eb
SUCCESS Python-bundle-PyPI-2023.10-GCCcore-13.2.0.eb
SUCCESS PyYAML-6.0.1-GCCcore-13.2.0.eb
SUCCESS bcrypt-4.1.3-GCCcore-13.2.0.eb
SUCCESS hypothesis-6.90.0-GCCcore-13.2.0.eb
SUCCESS expecttest-0.2.1-GCCcore-13.2.0.eb
SUCCESS Cython-3.0.10-GCCcore-13.2.0.eb
SUCCESS SciPy-bundle-2023.07-gfbf-2023a.eb
SUCCESS SciPy-bundle-2023.11-gfbf-2023b.eb
SUCCESS MPFR-4.2.1-GCCcore-13.2.0.eb (non-Python)
SUCCESS MPC-1.3.1-GCCcore-13.2.0.eb (non-Python)
SUCCESS gmpy2-2.1.5-GCC-13.2.0.eb

Build succeeded for 14 out of 14 (12 easyconfigs in total) i7150 - Linux Rocky Linux 8.9 (Green Obsidian), x86_64, AMD EPYC 7702 64-Core Processor (zen2), Python 3.8.17 See https://gist.github.com/Flamefire/9b46c20a82f46a2b0ac8ac796c797ba0 for a full test report.

$ grep -F 'pip check` completed' /tmp/easybuild-tmplog/easybuild-9v3ppgb0.log | wc -l
12

--> Works

Flamefire commented 2 months ago

--module-only build

Test report by @Flamefire

Overview of tested easyconfigs (in order)

SUCCESS SciPy-bundle-2022.05-foss-2022a.eb
SUCCESS SciPy-bundle-2023.02-gfbf-2022b.eb
SUCCESS Python-bundle-PyPI-2023.06-GCCcore-12.3.0.eb
SUCCESS Python-bundle-PyPI-2023.10-GCCcore-13.2.0.eb
SUCCESS PyYAML-6.0.1-GCCcore-13.2.0.eb
SUCCESS bcrypt-4.1.3-GCCcore-13.2.0.eb
SUCCESS gmpy2-2.1.5-GCC-13.2.0.eb
SUCCESS hypothesis-6.90.0-GCCcore-13.2.0.eb
SUCCESS expecttest-0.2.1-GCCcore-13.2.0.eb
SUCCESS Cython-3.0.10-GCCcore-13.2.0.eb
SUCCESS SciPy-bundle-2023.07-gfbf-2023a.eb
SUCCESS SciPy-bundle-2023.11-gfbf-2023b.eb

Build succeeded for 12 out of 12 (12 easyconfigs in total) i7150 - Linux Rocky Linux 8.9 (Green Obsidian), x86_64, AMD EPYC 7702 64-Core Processor (zen2), Python 3.8.17 See https://gist.github.com/Flamefire/9f33337e3272a6195a6409513152ea14 for a full test report.

$ grep -F 'pip check` completed' /tmp/easybuild-tmplog/easybuild-ibyka26o.log | wc -l
12

--> Works

Flamefire commented 2 months ago

Test report by @Flamefire

Overview of tested easyconfigs (in order)

FAIL (build issue) poetry-1.8.3-GCCcore-13.3.0.eb (partial log available at https://gist.github.com/Flamefire/6cd543ca489c67da855bb773983a9a20)

Build succeeded for 0 out of 1 (1 easyconfigs in total) n1604 - Linux RHEL 8.9 (Ootpa), x86_64, Intel(R) Xeon(R) Platinum 8470 (icelake), Python 3.8.17 See https://gist.github.com/Flamefire/eabd7daf1272cc47e34a7d2f03d623b2 for a full test report.

Flamefire commented 2 months ago

For some reason the build dependencies are still loaded or get loaded in the sanity check after this change. Poetry depends on hatchling and has a builddep to scicit-build and that installs a packaging version (23.1) which is incompatible with hatchling 1.24.2 (requires packaging>=23.2)

Ok I found the issue: sanity_check_load_module must be called to also clean up the builddeps.

I overwrite _sanity_check_step_extensions now for this. This also ensures that the extensions are initialized. Related PR: https://github.com/easybuilders/easybuild-framework/pull/4620

boegel commented 2 months ago

@Flamefire Although this is only a bug fix, please re-target this to the 5.0.x to only include it with EasyBuild 5.0.

I think you're reasoning makes sense (haven't checked the diff in detail yet), but I want to err on the side of caution with this one.

That maybe also opens the door to not bother deprecating things, not sure at this point.

Flamefire commented 2 months ago

Test report by @Flamefire

Overview of tested easyconfigs (in order)

SUCCESS SciPy-bundle-2022.05-foss-2022a.eb
SUCCESS SciPy-bundle-2023.02-gfbf-2022b.eb
SUCCESS Python-bundle-PyPI-2023.06-GCCcore-12.3.0.eb
SUCCESS Python-bundle-PyPI-2023.10-GCCcore-13.2.0.eb
SUCCESS PyYAML-6.0.1-GCCcore-13.2.0.eb
SUCCESS bcrypt-4.1.3-GCCcore-13.2.0.eb
SUCCESS gmpy2-2.1.5-GCC-13.2.0.eb
SUCCESS hypothesis-6.90.0-GCCcore-13.2.0.eb
SUCCESS expecttest-0.2.1-GCCcore-13.2.0.eb
SUCCESS Cython-3.0.10-GCCcore-13.2.0.eb
SUCCESS SciPy-bundle-2023.07-gfbf-2023a.eb
SUCCESS SciPy-bundle-2023.11-gfbf-2023b.eb
SUCCESS libarchive-3.7.4-GCCcore-13.3.0.eb
SUCCESS CMake-3.29.3-GCCcore-13.3.0.eb
SUCCESS Rust-1.78.0-GCCcore-13.3.0.eb
SUCCESS maturin-1.6.0-GCCcore-13.3.0.eb
SUCCESS cffi-1.16.0-GCCcore-13.3.0.eb
SUCCESS cryptography-42.0.8-GCCcore-13.3.0.eb
SUCCESS poetry-1.8.3-GCCcore-13.3.0.eb

Build succeeded for 19 out of 19 (13 easyconfigs in total) i7078 - Linux Rocky Linux 8.9 (Green Obsidian), x86_64, AMD EPYC 7702 64-Core Processor (zen2), Python 3.8.17 See https://gist.github.com/Flamefire/b950ce066f48dfe5ff70e6286ef5604d for a full test report.

Flamefire commented 2 months ago

Test report by @Flamefire

Overview of tested easyconfigs (in order)

SUCCESS SciPy-bundle-2022.05-foss-2022a.eb
SUCCESS SciPy-bundle-2023.02-gfbf-2022b.eb
SUCCESS Python-bundle-PyPI-2023.06-GCCcore-12.3.0.eb
SUCCESS Python-bundle-PyPI-2023.10-GCCcore-13.2.0.eb
SUCCESS PyYAML-6.0.1-GCCcore-13.2.0.eb
SUCCESS bcrypt-4.1.3-GCCcore-13.2.0.eb
SUCCESS gmpy2-2.1.5-GCC-13.2.0.eb
SUCCESS hypothesis-6.90.0-GCCcore-13.2.0.eb
SUCCESS expecttest-0.2.1-GCCcore-13.2.0.eb
SUCCESS Cython-3.0.10-GCCcore-13.2.0.eb
SUCCESS SciPy-bundle-2023.07-gfbf-2023a.eb
SUCCESS SciPy-bundle-2023.11-gfbf-2023b.eb
SUCCESS poetry-1.8.3-GCCcore-13.3.0.eb

Build succeeded for 13 out of 13 (13 easyconfigs in total) i7012 - Linux Rocky Linux 8.9 (Green Obsidian), x86_64, AMD EPYC 7702 64-Core Processor (zen2), Python 3.8.17 See https://gist.github.com/Flamefire/55d0fd72b37f71a1fac528cca182621a for a full test report.

Flamefire commented 2 months ago

That maybe also opens the door to not bother deprecating things, not sure at this point.

So what shall I do? I'd go with keeping the deprecation calls such that it will be an error immediately in 5.0x

I have some EC PRs to go with this:

Because I noticed that at some places ext_default_options is used instead of the PythonBundle options which triggers this check and just did a search for such ECs. I'd merge those once the test reports are in and then check the 5.0x branch too. If we don't we'll likely get new ECs created from those reintroducing the "issue" and the change is pretty straight forward and easy to even manually verify.

Flamefire commented 2 months ago

Opened a PR against 5.0x: https://github.com/easybuilders/easybuild-easyblocks/pull/3432
Or would you prefer to update this PR?

easybuilders / easybuild-easyblocks