feat: build[uv] - Githubissues

henryiii commented 3 weeks ago

This adds support for setting the build backend to build[uv], which will use the build backend with the uv installer option. Using this will also move all installs over to uv if possible. Python <3.8 will still use pip.

There's one workaround we need to fix first; uv doesn't seem to support URI's. Edit: fixed upstream in https://github.com/astral-sh/uv/pull/4145, but we pulled our existing workaround out into a function, so going with that for now, can be updated once a new uv is out and common.

This just uses the system uv everywhere (except on linux), which is fastest and gives the user control when installing it.

Also checking the tests by setting this as the default in the CI sample job; this actually built without isolation before, so it's a little slower, but much faster than if we enabled isolation any other way. Our build tests use it, which actually is faster, except for a few that expected pip to be present. The uv backend doesn't need to be backward compatible with anything, so it installs as little as possible (also faster).

Build example at https://github.com/henryiii/deflate/actions/workflows/build.yml. Speedup is visible in the images below.

`pip`	`build[uv]`

henryiii commented 3 weeks ago

Opened an issue about missing URI support in uv: https://github.com/astral-sh/uv/issues/4124

henryiii commented 3 weeks ago

I marked this ready for review. I think this + #1854, combined with Pyodide, would be a good new minor release.

I chose build[uv] because a) you can't use uv + pip to build, so enabling uv but leaving pip as the builder would lose part of the speedup, b) installing pip wastes time, c) eventually you might be able to use uv directly, which would be a new uv frontend option that would fit in nicely, and d) I want more people using the build frontend. :) Open to ideas, though, if anyone prefers other things.

henryiii commented 3 weeks ago

Even just moving some (not all!) of the build tests and none of the pip tests still makes a noticeable impact on our test time:

Job	Before	This PR
Windows	46 m	40 m
Linux 3.12	39 m	38 m
Linux 3.8	36 m	32 m
macOS 13	28 m	24 m
macOS 14	14 m	11 m

mayeut commented 3 weeks ago

I cancelled the Travis CI build to get manylinux to build. I'll review this and restart the Travis CI build (I'm especially interested in seeing what happens on ppc64le with this PR given https://github.com/pypa/manylinux/pull/1621 fails but running with qemu just works....)

mayeut commented 3 weeks ago

This just uses the system uv everywhere (except on linux), which is fastest and gives the user control when installing it.

I agree with both pros you're mentioning but I'd just like to mention some cons (I might be wrong, please do correct me):

The user needs to take some steps to install uv if they want to use the feature. (it's easy enough to install but still)
uv seems to be evolving quite fast and may introduce some unforseen breaking changes, using a pinned version that's being managed in cibuildwheel cache would allow not to break on those. (in this case, we'd nevertheless need to check if multiple versions of uv can co-exist - I'm mostly thinking about its own cache here).

That being said, I'm fine with the current state and we can decide later on if this needs to be updated.

Am I wrong in thinking there are no tests for macOS tests with this new frontend ? uv being platform specific, I wonder what happens while testing x86_64 on arm64. ~~I will do a quick test and report back.~~

~~Edit: it just works even though uv is not universal2.~~ Edit 2: my test was buggy, the linter caught it. It is failing with:

arch: posix_spawnp: uv: Bad CPU type in executable
Error: Command ['arch', '-x86_64', 'uv', 'venv', '/private/var/folders/hb/2n_7f3yn20v06lzz3129dtw80000gn/T/cibw-run-1_t80lhb/cp312-macosx_universal2/venv-test-x86_64', '--python=python'] failed with code 1. None

henryiii commented 3 weeks ago

Does it work if we don't add the 'arch', '-x86_64'? That is, can uv target an Intel Python from ARM? (I'm asking in Astral's Discord)

Actually, I think --python-platform x86_64-apple-darwin should do it (when installing).

mayeut commented 3 weeks ago

Actually, I think --python-platform x86_64-apple-darwin should do it (when installing).

It's working locally, let's see how it goes in CI.

mayeut commented 3 weeks ago

can uv target an Intel Python from ARM?

I don't know what you asked exactly but my guess would be yes - even though it might not be tested. It's worth noting that using the intel64 suffixed binaries, uv pip install works without adding --python-platform x86_64-apple-darwin Using those binaries would require to change the way the test virtual environment is created and would allow the user to use some uv commands in CIBW_BEFORE_TEST, CIBW_TEST_COMMAND without hassle but... free-threaded python does not provide this binary (we could probably ask if we think it's worth it):

cibuildwheel %  find /Library/Frameworks/Python*.framework/Versions -name '*-intel64'
/Library/Frameworks/Python.framework/Versions/3.10/bin/python3-intel64
/Library/Frameworks/Python.framework/Versions/3.10/bin/python3.10-intel64
/Library/Frameworks/Python.framework/Versions/3.11/bin/python3-intel64
/Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11-intel64
/Library/Frameworks/Python.framework/Versions/3.13/bin/python3-intel64
/Library/Frameworks/Python.framework/Versions/3.13/bin/python3.13-intel64
/Library/Frameworks/Python.framework/Versions/3.12/bin/python3-intel64
/Library/Frameworks/Python.framework/Versions/3.12/bin/python3.12-intel64
/Library/Frameworks/Python.framework/Versions/3.9/bin/python3-intel64
/Library/Frameworks/Python.framework/Versions/3.9/bin/python3.9-intel64
/Library/Frameworks/Python.framework/Versions/3.8/bin/python3-intel64
/Library/Frameworks/Python.framework/Versions/3.8/bin/python3.8-intel64

cibuildwheel % find /Library/Frameworks/PythonT.framework/Versions/3.13/bin/
/Library/Frameworks/PythonT.framework/Versions/3.13/bin/
/Library/Frameworks/PythonT.framework/Versions/3.13/bin//python3.13t
/Library/Frameworks/PythonT.framework/Versions/3.13/bin//python3.13t-config
/Library/Frameworks/PythonT.framework/Versions/3.13/bin//pip3.13
/Library/Frameworks/PythonT.framework/Versions/3.13/bin//pip3
/Library/Frameworks/PythonT.framework/Versions/3.13/bin//pip
/Library/Frameworks/PythonT.framework/Versions/3.13/bin//python3.13

mayeut commented 3 weeks ago

I chose build[uv] because a) you can't use uv + pip to build, so enabling uv but leaving pip as the builder would lose part of the speedup, b) installing pip wastes time, c) eventually you might be able to use uv directly, which would be a new uv frontend option that would fit in nicely, and d) I want more people using the build frontend. :) Open to ideas, though, if anyone prefers other things.

I might do a) in my fork just to see the improvement we'd get. Regarding d), we might want to change the default backend soon ?

henryiii commented 3 weeks ago

but... free-threaded python does not provide this binary (we could probably ask if we think it's worth it)

Probably could ask @ned-deily for the intel suffixed free-threading, it would be nice if user could just use uv directly with the environment we give them, rather than always having to add the platform target flag. Also wonder why there's no python3t, only python3.13t, though it doesn't really affect us, we can always add the version number.

Regarding a), we probably could use uv automatically for all the installs we do (including pip), though we can't rely on it, so it's strictly a speed up, not a simplification. I don't think we can move any user installs implicitly to uv, since uv isn't a guaranteed drop-in replacement; things like dependencies that require dev versions will fail with uv. But on things we control, I think it's safe to use it if present. But we could always do that later. We can also use --no-compile, which is uv's default but might speed pip up a little.

That's one nice benefit of this: build[uv] is an opt-in without backwards compatibilty, but it's an attractive opt-in (speed is a good carrot!) and we can get better exposure of our build option before a default change. It's also a chance to remove back-compat pre-installs and give the user a completely free and clean environment.

Regarding d), this would increase the usage of the build backend, and moving the default (to build without uv? The fact it's a little slower than pip due to less setup is annoying, though, without uv; using uv for our internal installs would fix that, I think) would be something interesting, maybe even for a 3.0 release.

ned-deily commented 3 weeks ago

Sure, we can add both of those additional names. Please open an issue for that on the cpython tracker.

mayeut commented 3 weeks ago

There's one last item before this can be merge I think. Shall the action:

provide a new input to request uv to be installed ?
just install uv unconditionally ?
or do we expect it to be pre-installed by the user ?

njzjz commented 3 weeks ago

Got the following error when testing a universal2 wheel:

Testing wheel on x86_64...

  + uv venv /private/var/folders/lr/439_fwvd3m76p9vy50d57kcc0000gn/T/cibw-run-jurr1zjt/cp39-macosx_universal2/venv-test-x86_64 --python=python
  Using Python 3.9.13 interpreter at: /private/var/folders/lr/439_fwvd3m76p9vy50d57kcc0000gn/T/cibw-run-jurr1zjt/cp39-macosx_universal2/build/venv/bin/python
  Creating virtualenv at: /private/var/folders/lr/439_fwvd3m76p9vy50d57kcc0000gn/T/cibw-run-jurr1zjt/cp39-macosx_universal2/venv-test-x86_64
  Activate with: source /private/var/folders/lr/439_fwvd3m76p9vy50d57kcc0000gn/T/cibw-run-jurr1zjt/cp39-macosx_universal2/venv-test-x86_64/bin/activate
  + arch -x86_64 which python
  /private/var/folders/lr/439_fwvd3m76p9vy50d57kcc0000gn/T/cibw-run-jurr1zjt/cp39-macosx_universal2/venv-test-x86_64/bin/python
  + /Users/runner/.cargo/bin/uv pip install --python-platform x86_64-apple-darwin /private/var/folders/lr/439_fwvd3m76p9vy50d57kcc0000gn/T/cibw-run-jurr1zjt/cp39-macosx_universal2/repaired_wheel/mddatasetbuilder-0.1.dev1+g08f3b70-cp37-abi3-macosx_10_9_universal2.whl
  Resolved 30 packages in 227ms
  error: Failed to download distributions
    Caused by: Failed to fetch wheel: matplotlib==3.9.0
    Caused by: Failed to build: `matplotlib==3.9.0`
    Caused by: Build backend failed to build wheel through `build_wheel()` with exit status: 1
  --- stdout:
  meson-python: error: Multi-architecture builds are not supported but $ARCHFLAGS='-arch arm64 -arch x86_64'
  --- stderr:

  ---
                                                              ✕ 4.90s
Error: Command ['/Users/runner/.cargo/bin/uv', 'pip', 'install', '--python-platform', 'x86_64-apple-darwin', '/private/var/folders/lr/439_fwvd3m76p9vy50d57kcc0000gn/T/cibw-run-jurr1zjt/cp39-macosx_universal2/repaired_wheel/mddatasetbuilder-0.1.dev1+g08f3b70-cp37-abi3-macosx_10_9_universal2.whl'] failed with code 2. None

mayeut commented 3 weeks ago

Thanks for the report @njzjz, int can be reproduced in tests, I'll commit the test failure. I'll propose a fix later today.

mayeut commented 3 weeks ago

I don't know why the updated test does not fail in CI.

Edit: I can't build pillow from sources locally but CI can.

mayeut commented 3 weeks ago

uv tries to download/install wheels compatible with MACOSX_DEPLOYMENT_TARGET when --python-platform x86_64-apple-darwin is used. This is a feature, not a bug. matplotlib or pillow do not provide 10.9 wheels so uv tries to build them from sources which fails.

MACOSX_DEPLOYMENT_TARGET is inherited from the build environment (as well as other environment variables that should only be useful when building).

Rather than just removing this environment variable from the test environment, I propose we use the current version where tests are running. It could be useful in the edge case where a native dependency needs to be built in order to run the tests.

henryiii commented 3 weeks ago

IMO, I think it would be better to remove MACOSX_DEPLOYMENT_TARGET in the tests environment, as that is more likely to mimic what a user will have, and be a bit less surprising overall. Libraries should still build, though users should hopefully not be building binaries in the test step. In general, it would be nice if we could validate this - a library should not build for a lower macOS deployment target than its dependencies. Unfortunately, validating it is hard, as test dependencies might not count (could be optional dependencies), for example.

I thought about the action a bit, I think it's best to just expect someone to setup uv in a previous step for now. I've been using

      - name: Setup uv
        uses: yezz123/setup-uv@v4

      - name: Build wheels
        uses: henryiii/cibuildwheel@henryiii/feat/builduv

Longer term, maybe the action could check cibuildwheel's config and detect if build[uv] was present via any env var or config setting?

mayeut commented 3 weeks ago

After reading a bit more on the usage of MACOSX_DEPLOYMENT_TARGET with --python-platform x86_64-apple-darwin in uv, not setting it will lead to 12.0 being used.

I'd rather set MACOSX_DEPLOYMENT_TARGET for now in order not to hit this limitation if someone depends on a 13.0/14.0 wheel.

Once we can use -intel64 suffixed binaries, we can remove all of those build specific environment variables.

I thought about the action a bit, I think it's best to just expect someone to setup uv in a previous step for now. Longer term, maybe the action could check cibuildwheel's config and detect if build[uv] was present via any env var or config setting?

This plan sounds good.

it would be nice if we could validate this - a library should not build for a lower macOS deployment target than its dependencies. Unfortunately, validating it is hard, as test dependencies might not count (could be optional dependencies), for example.

Well, while I do agree with general idea, I think probably not doable and there always will be someone that does want to use an up-to-date version of something with outdated versions of dependencies (maybe just by using -only-binary), there are tests dependencies that might not count, there's abi3 and the fact that CPython 3.13 is bumping its deployment target (thinking I use ABI3 target 10.9 and I depend on e.g. numpy, which will be 10.13 once on Python 3.13) and probably much more I'm not thinking about.

henryiii commented 3 weeks ago

not setting it will lead to 12.0 being used.

Okay, sounds good as is then.

I think probably not doable

That’s what I meant by hard. At best we might come up with a way to add a warning.

Good to go then?

pypa / cibuildwheel

feat: build[uv] #1856