[infra] Upgrade Python to 3.10.14 in base-builder & base-runner Images

DaveLak commented 1 month ago

[!NOTE]
I was looking for somewhere to get feedback from maintainers about this approach to the Python 3.10 upgrade before attempting it, but the discussion surrounding a Python upgrade has been rather fragmented across many issues, PRs, and comment chains.

For that reason, I felt it would be easier to propose with a working example and dedicated PR.

Fixes:

11419
9638

Supersedes:

9532
11420

Changes

The changes introduced here upgrade Python from 3.8 to 3.10.14 inside the base-builder and base-runner images.

Base Image Changes

Image	Before Changes	After Changes
base-builder	Compiled Python 3.8 from source using official release servers at https://www.python.org/ftp/python/.	Compiles Python 3.10.14 (the latest 3.10 release) from source using official release servers at https://www.python.org/ftp/python/.
base-runner	Installed Python 3.8 from the default apt repository provided by the Ubuntu 20.04 image.	Uses a multi-stage build to copy the Python 3.10.14 interpreter compiled by the base-builder image, ensuring version sync and saving build time by re-using a pre-built version.

Known Impact on Projects

3.9 Workarounds That Can Be Removed

Project	Fix Link
dask	https://github.com/DaveLak/oss-fuzz/commit/417bbf566e3ffd962e09a50903282233b3676b69
docutils	https://github.com/DaveLak/oss-fuzz/commit/e4c21ffbd8baa83d97d7dd0e1f8e3e7adeae43d1
dovecot	https://github.com/DaveLak/oss-fuzz/commit/7ab3ab6e2604396f99398a1236ff6ddb91b04430
nbclassic	https://github.com/DaveLak/oss-fuzz/commit/5509b4e221ecad6136b353780daf1ceb3f457f53
pandas	https://github.com/DaveLak/oss-fuzz/commit/0642a7afa96d193439cc9ebd7659a0fccc7e5c1b
pybind11	https://github.com/DaveLak/oss-fuzz/commit/a5bbdb3dfbddd7f37dd7737cb57bb39c9ec5cc5f
pyodbc	https://github.com/DaveLak/oss-fuzz/commit/afa2b5edc96d581329fe0405d463b026abbea3ad
qpid-proton	https://github.com/DaveLak/oss-fuzz/commit/f5bf75628649fea5696fd8e95e1e698092ea4e74

Anticipated Build Failures

Preexisting Failures

Fix is Prepared

Project	Fix Link
airflow	https://github.com/DaveLak/oss-fuzz/commit/60a03686280b63e88773ef9b25cd191ab8e959d0
ipython	https://github.com/DaveLak/oss-fuzz/commit/21ac68e9ba44e063d04b3df394330f6490081bcb
networkx	https://github.com/DaveLak/oss-fuzz/commit/fc2f8c5481262849b586ebb8afdfdf3e6672f44f
numpy	https://github.com/DaveLak/oss-fuzz/commit/9383c876dccad016f169d43a948c5bdc45b2894d
tensorflow-addons	https://github.com/DaveLak/oss-fuzz/commit/eed2bea1190abd51f3c621ccc923fecb6585a67c
django (coverage build)	https://github.com/DaveLak/oss-fuzz/commit/c724d61eae52e309726dc0d90f22bdbbecee04e7
proto-plus-python	https://github.com/DaveLak/oss-fuzz/commit/37d973ee8726c33678da37fb41e48d264418f097
dnspython	The upgraded pip version in the base-builder fixes the currently failing build.

Fix Requires Upstream Changes

Project	Issue
pyvex	Currently failing on python 3.9 because `archinfo` dependency requires >=3.10. Fails after the 3.10 upgrade because the upstream build script needs `python3.9` replaced with `python3`.

Requires More Investigation

Project	Issue
matplotlib	Upgrading Python & Pyinstaller does resolve the build issues, but an error in the fuzz harness is exposed and must be resolved for check_build to pass. The exception: `TypeError: Parser.non_math() takes 2 positional arguments but 4 were given" in "File "fuzz_plt.py", line 43, in TestOneInput`.
scipy	Upgrading Python & Pyinstaller does resolve the build issues, but an error in the build step causes the build to fail. The error seems related to the linking: "/usr/bin/ld: /usr/bin/ld: DWARF error: invalid or unhandled FORM value: 0x25". When `export LDFLAGS="-fuse-ld=lld"` is set, the error becomes: "`ld.lld: error: undefined symbol: __asan_report_store4`".
pandas (Introspector only)	This workaround in `build.sh` is the issue.
pycrypto	Failing with error: "`SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats`". Seems like the issue described here. Pycrypto is deprecated and this is unlikely to be fixed upstream.

Possible Future Improvements

Using the base-builder image in a multi-stage build to copy the pre- compiled Python into base-runner is effective, but feels like a workaround that may be introducing tech debt. A cleaner approach would be to extract the Python compilation into a discrete base image similar to how base-clang works, and use that as the multi-stage builder in images that need it.

Fuzz Introspector Caveat

Fuzz Introspector currently uses Python 3.9. While an upgrade to 3.10 is not expected to introduce any new issues, it was not tested on these changes and may require additional work.

Motivation

Python 3.8 is reaching end of life in October 2024.
The Scientific Python Community already encourages dropping 3.8 support.
- This is evident when looking at which projects have resorted to upgrading to newer Pythons using ad-hoc workarounds (see numpy, scipy, pandas, etc.)
It is likely that more Python projects will begin dropping support for 3.8, further increasing the number of broken builds and ad-hoc workarounds.
Code coverage does not work on Python projects that use Python 3.10+ syntax.
Previous attempts at upgrading Python have stalled (see https://github.com/google/clusterfuzz/issues/3290#issuecomment-1699573706 & the issues linked under "Supersedes" above.)
- In recognition of the fact that OSS-Fuzz maintainers are stretched thin, I thought I'd give it a shot.

jonathanmetzman commented 3 weeks ago

/gcbrun trial_build.py all --sanitizer coverage address --fuzzing-engine libfuzzer

jonathanmetzman commented 2 weeks ago

/gcbrun trial_build.py all --sanitizer coverage address --fuzzing-engine libfuzzer

DaveLak commented 2 weeks ago

Thanks for the runs. I'll check the timeouts in about 24 hours from now.

DaveLak commented 2 weeks ago

@jonathanmetzman I think e1a6e9fe59e29563f0a5dfbaeecd9d8c71762fdd should fix the broken coverage builds.

jonathanmetzman commented 5 days ago

/gcbrun trial_build.py all --sanitizer coverage address --fuzzing-engine libfuzzer

DonggeLiu commented 2 days ago

google / oss-fuzz