Allow building CPython from a local source directory

[zanieb]

As a first step towards https://github.com/indygreg/python-build-standalone/issues/139, adds the ability to build from a local source directory instead of downloading CPython from python.org via a new --python-source <path> build option.

e.g. tested by cloning CPython at tag v3.12.1 then building

PYBUILD_PYTHON_VERSION=3.12.1 python3.12 ./build-macos.py \
    --target-triple aarch64-apple-darwin \
    --python cpython-3.12 --optimizations debug \
    --python-source $(pwd)/../cpython

I also tested this with the latest commit on the CPython 3.12 branch.

There are a lot of assumptions about build information being available in the DOWNLOADS file, which makes this a little brittle. I also only implemented this for Unix.

I tried to make as minimal of a change as possible i.e. instead of refactoring to allow an unpacked source directory to be passed around we put the given source in an archive simulating a download. We could follow this with some more changes to abstract Python source downloads.

Additional safeguards could be added to avoid common mistakes, like ensuring that:

• The PYBUILD_PYTHON_VERSION version is compatible with the --python version
• The --python-source path exists
• Relative --python-source paths are resolved based on the original invocation working directory

Additionally, we could improve usability by:

• Inferring the PYBUILD_PYTHON_VERSION from the source files
• Allowing the source to be a remote path e.g. from GitHub
• Read commit information from Git and include in builds

Next steps for testing in-progress CPython builds would involve:

• Setting up a CI job to pull and build from Python 3.12's development branch
• Initial Python 3.13 support
• Similar changes to the Windows builds

[zanieb]

For my own bookkeeping... failed with the following on Linux — I'll test that out locally and fix.

Traceback (most recent call last):
  File "/home/runner/work/python-build-standalone/python-build-standalone/cpython-unix/build.py", line 1217, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/runner/work/python-build-standalone/python-build-standalone/cpython-unix/build.py", line 1199, in main
    build_cpython(
  File "/home/runner/work/python-build-standalone/python-build-standalone/cpython-unix/build.py", line 739, in build_cpython
    build_env.install_toolchain_archive(
TypeError: ContainerContext.install_toolchain_archive() got an unexpected keyword argument 'version'
make: *** [Makefile:311: /home/runner/work/python-build-standalone/python-build-standalone/build/cpython-3.9.18-mips-unknown-linux-gnu-lto.tar] Error 1

[zanieb]

@indygreg I'm blocked by #238 on verifying that we can use this new flag in Linux, but I made sure everything works in CI in my fork.

A potential alternative worth considering is using the docker -v functionality to bind mount the source checkout into the build environment. You would need to hack up build-cpython.sh to copy the source files into the /build directory. I think always normalizing to a source archive makes a lot of sense.

Yeah the normalization makes sense to me for now. It's not really a bottleneck.

I agree it would be nice to have smarter version resolution or Git (both local and remote repo) integration. But those could be added later.

I can open issues to track those.

I'm sure you figured this out, but the build-main.py script and the way we use PYBUILD_* environment variables to thread state around is super hacky. Evolved complexity. If you have ideas for simplifying, I'm all ears.

I'm not ready to make any drastic changes, but I'll keep it in mind. I actually didn't think it was too bad.

Maybe we could integrate rye or uv to make bootstrapping the build environment simpler?

Honestly I was impressed by your bootstrapping :) I'd definitely be willing to add uv though.

[zanieb]

Just to check in here again, is there something you are looking for before this can be merged? Should I just stack another pull request that starts to make use of this? Since main is moving slow I don't mind building up a stack showing functionality before merging.