feat: add a uv backend

[henryiii]

UV just came out from the Ruff folks, and it's insanely fast - it makes venvs faster (25ms) than Python can start up (50ms). And much faster than virtualenv (400ms) and venv (nearly 5 seconds). The package install speeds are also unreal compared to pip.

There are some differences - namely name @ . is required instead of . if an install is not editable (for now). But it's really nice to try it out!

https://github.com/wntrblm/nox/assets/4616906/1fe4c981-9d9a-4aaf-b19e-94c79daeebe7

Nox's docs build in 4 seconds instead of 22 seconds using the uv backend.

[henryiii]

I pulled your changes in here, thanks @alex! (Nice username, by the way!)

[alex]

Happy to help! Looking forward to this.

[henryiii]

I'm thinking a nice followup could be to allow fallback backends. So uv|virtualenv would use UV if available, and fallback to virtualenv if not available. Same with mamba|conda.

[henryiii]

Is there a way to ignore a line from coverage only on Python 3.7?

[edgarrmondragon]

Is there a way to ignore a line from coverage only on Python 3.7?

Maybe https://github.com/asottile/covdefaults?tab=readme-ov-file#version-specific--pragma-no-cover?

[alex]

I don't believe there's a way to skip coverage by python version with just coverage.py. I think you have two choices:

1) #pragma: no cover the line always 2) Install uv outside the virtualenv, with the system python interpreter, so that it's on the $PATH in CI.

[henryiii]

@FollowTheProcess, @DiddiLeija, @cjolowicz, or @theacodes, I think this is ready. Also, there are quite a few other PRs that are basically ready and useful to have open too!

[henryiii]

Also, Tox added UV support yesterday: https://github.com/tox-dev/tox-uv

[slafs]

Fantastic work! Thank you all! My 3 cents:

1. [minor] The uv option is not mentioned in the help text of --default-venv-backend nor --force-venv-backend CLI options.
2. It's also not mentioned in the docs for those options ("Changing the sessions default backend" and "Forcing the sessions backend" in usage.rst).
3. Speaking of docs, would it make sense to add a note that uv venv does NOT install pip by default? Unlike venv or virtualenv. I've run into a (rather obscure) caveat when trying out this branch. Namely you can't switch backends from uv (fresh) to venv (re-use) without re-creating the whole virtual environment, because pip is missing. It's OK to go from venv (fresh) to uv (re-use), though. Again, this is probably a non-issue in a day-to-day usage and only relevant when exploring backends or seeing that ridiculous speed difference 😅. Nevertheless, I feel like it wouldn't hurt to mention that in the docs.

Also, thanks @skshetry for pointing out that bug in uv 😬.

I'm thinking a nice followup could be to allow fallback backends. So uv|virtualenv would use UV if available, and fallback to virtualenv if not available. Same with mamba|conda.

Yes! 💯 That would be great!

[henryiii]

I kind of forgot about docs because I was initially just making it work, will add!

[FollowTheProcess]

@henryiii nice work! Sorry been a bit swamped recently and haven't had much time to look at Nox too much. I think once the docs get an update calling out the "no pip" behaviour as mentioned above then I'm happy and excited to get this in 👍🏻

[henryiii]

@slafs: Thanks! I've addressed (1) and (2). I think I also fixed (3) by detecting uv created venvs, would be great if you could check!

[henryiii]

CI is busted on Python 3.7 until https://github.com/conda-forge/conda-forge-repodata-patches-feedstock/pull/667 goes in. Edit: fix in, should roll out in about an hour. Edit: I think it's out.

[henryiii]

CI should be restartable now, fix looks like it's available.

[henryiii]

LGTM, but I don't think (3) is fixed, right?

(3) should be "fixed" in the case that if you switch environments, it should rebuild the environment now, as it can recognize that the environment was created by virtualenv or uv. You won't have pip inside the uv environment unless you install it, like any other dependency. But .install(...) should always work.

(If not, that's likely a bug!)

[slafs]

Hmmm... I might be holding it wrong, but with this noxfile.py:

import nox

@nox.session()
def xxx(session):
    session.install("cowsay")
    session.run("cowsay", "-t", "Hello world")

running:

nox -N -db uv -s xxx

and then:

nox -r -s xxx

gives me:

nox > Running session xxx
nox > Re-using existing virtual environment at .nox/xxx.
nox > python -m pip install cowsay
nox > Command python -m pip install cowsay failed with exit code 1:
/private/tmp/noxuvtest/.nox/xxx/bin/python: No module named pip
nox > Session xxx failed.

I think reuse_existing_virtualenvs option is the key here (we have it on by default in our noxfile - via global nox.options.reuse_existing_virtualenvs = True). AFAIU without this option nox always recreates the environment, no? 🤔

[henryiii]

Hmm, that's exactly what I thought should work now, it should recreate the environment since the backend changed. I'll investigate.

[henryiii]

Ah, I think it only recreates if NOX_ENABLE_STALENESS_CHECK is in the environment. If the environment type changes or the interpreter is different, then it rebuilds, but only if this environment variable is set.

This seems a little weak - if the environment type changes or the interpreter changes are fairly rare and pretty big changes that I'd assume you'd want always want to rebuild for? "Staleness" to me would be time based or package based, something other than changing the venv type or the interpreter. Moving from venv to conda, for example, would also be an error.

But that's why it's not rebuilding.

[layday]

Feel like nox should either error out or recreate the venv if the active venv backend != backend the venv was created with. A marker file in the venv folder might work, or reading pyvenv.cfg - both virtualenv and uv add a key with the backend's name. I don't know about conda and mamba.

[cjolowicz]

This is a known issue. That environment variable is a feature flag for some incomplete work to fix it, and was never publicly announced. I think this PR is good as is.

[cjolowicz]

Could you re-base this?

[henryiii]

Requiring 100% coverage for each job is causing problems, the tox 3 jobs are not running the tox 4 parts. The total coverage is fine, but not the per job coverage.

[henryiii]

And the "passing" tox 4 one looks very suspicious!

Edit, ahh, it's skipped on Python 3.7, so that's expected. So this probably would fail for both 3 and 4.

[cjolowicz]

Requiring 100% coverage for each job is causing problems, the tox 3 jobs are not running the tox 4 parts. The total coverage is fine, but not the per job coverage.

This is unfortunate, we shouldn't require 100% coverage for individual jobs. We should only gather the coverage data in those jobs. One way to achieve this behavior is to have a separate coverage report session and only notify it when session.interactive is true. But I see you're already on to this. :+1:

[henryiii]

Yeah, it looks like we did and I broke it somehow. Going to try to revert the coverage changes (in a few hours probably)

[henryiii]

I can remove the covdefaults & conditional pragmas, they aren't needed, it's being checked at the end again.

[cjolowicz]

I'd be happy for you to reset to 9c935699987a04bf8a96f6bc04ba50927b903d9f and clean up the coverage-related code in a follow-up PR if that's easier.

[henryiii]

I've pulled out the fix #778 (which is then running into issues merging due to Windows paths, I think). Currently there are a lot of "no covers" due to this on different OSs, and I'd like to not have to add more for the new code.

[henryiii]

I'll rebase on #778 and remove coverage changes when that goes in.

[henryiii]

Thanks everyone!

[alex]

Thank you for following through on this!

Is there a good place to follow along in terms of when this will be in a release?

[henryiii]

Watch -> Custom -> releases (in the GH UI) is what I do. You can also get GitHub Releases via RSS, though I don't use that much anymore.

I think we have a few small followups (rebuild on env change & environment fallback, maybe #776), and we might want #768 as I'd like the action to work on the new AS runners, but after that I think @theacodes has said that a release is in the near future.

[alex]

Thanks

On Fri, Feb 23, 2024 at 12:19 PM Henry Schreiner @.***> wrote:

Watch -> Custom -> releases (in the GH UI) is what I do. You can also get GitHub Releases via RSS, though I don't use that much anymore.

I think we have a few small followups (rebuild on env change & environment fallback, maybe #776 https://github.com/wntrblm/nox/issues/776), and we might want #768 https://github.com/wntrblm/nox/pull/768 as I'd like the action to work on the new AS runners, but after that I think @theacodes https://github.com/theacodes has said that a release is in the near future.

— Reply to this email directly, view it on GitHub https://github.com/wntrblm/nox/pull/762#issuecomment-1961707774, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAGBBJ6VK4G7NC3RAIG4TYVDFSPAVCNFSM6AAAAABDMYCT5KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNRRG4YDONZXGQ . You are receiving this because you were mentioned.Message ID: @.***>

-- All that is necessary for evil to succeed is for good people to do nothing.

[henryiii]

FYI, release is out (including on PyPI and homebrew). I've been setting NOX_DEFAULT_VENV_BACKEND=uv and it's so fast. It's also teaching me where I unknowingly had dependencies on pip and need to list it explicitly.

After the new release is out for a bit I may start moving over to the new uv|virtualenv syntax.

[alex]

Thank you!

On Mon, Mar 4, 2024 at 11:48 AM Henry Schreiner @.***> wrote:

FYI, release is out (including on PyPI and homebrew). I've been setting NOX_DEFAULT_VENV_BACKEND=uv and it's so fast. It's also teaching me where I unknowingly had dependencies on pip and need to it listed explicitly.

After the new release is out for a bit I may start moving over to the new uv|virtualenv syntax.

— Reply to this email directly, view it on GitHub https://github.com/wntrblm/nox/pull/762#issuecomment-1977028752, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAGBE2O77KAZL5FHD3TNLYWSQWZAVCNFSM6AAAAABDMYCT5KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNZXGAZDQNZVGI . You are receiving this because you were mentioned.Message ID: @.***>

-- All that is necessary for evil to succeed is for good people to do nothing.

[pfmoore]

Looking at this feature now it's been released, I have a couple of comments.

1. The uv backend isn't mentioned in the documentation of the session API. It took me a while to find the information in the "Usage" section.
2. While it's noted that uv is fast, and that it doesn't install pip by default, it's not obvious from the way that the documentation covers this, that a significant part of the speed gain is because pip isn't installed. By adding a venv_params argument of --without-pip (for venv or virtualenv) you get much of the speed gain (along with the "pip isn't installed" disadvantage, of course), while still keeping the (currently) more compatible standard behaviour otherwise.

[cjolowicz]

I've been wondering if we shouldn't support using an external pip in a more seamless way.

[theacodes]

I'd be fine with that.

On Wed, Mar 6, 2024 at 9:42 AM Claudio Jolowicz @.***> wrote:

I've been wondering if we shouldn't support using an external pip in a more seamless way.

— Reply to this email directly, view it on GitHub https://github.com/wntrblm/nox/pull/762#issuecomment-1981023447, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAB5I47DKXZ73HFKRFETVGDYW4TNLAVCNFSM6AAAAABDMYCT5KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBRGAZDGNBUG4 . You are receiving this because you were mentioned.Message ID: @.***>

[cjolowicz]

That said, IIUC every session.install would incur pip startup time twice, right?

[pfmoore]

That said, IIUC every session.install would incur pip startup time twice, right?

Does the uv backend use uv for session.install as well? That's not obvious from the description, where it's described as being a venv backend, not an installer backend. Although thinking about it now, it seems very reasonable to be that it's actually an "environment" backend in the broader sense of creating the environment and populating it (I've never used the conda backend, but I guess that works similarly).

I've been wondering if we shouldn't support using an external pip in a more seamless way.

Assuming there's a pip available in the environment nox is installed in, and it's sufficiently recent, you could easily use that pip (with the --python command line option) to do installs into the session. At that point, using --without-pip by default in virtualenv/venv created environments is likely to be possible without too much disruption.

I'd be willing to look at putting together a PR to do this, if there's interest. I've not contributed to the nox codebase before, so advice (in particular on the best user interface) would be welcome.

[henryiii]

Build 1.1 uses external pip if it can automatically. But of course, nox users can interact with pip. I’m wondering if the following would be seamless enough:

• Don’t install pip if pip is present and new enough in nox’s environment.

• Special case the installer (pip or uv) if it’s the first arg. Convert it to an absolute path and add any required args. I think conda does something like this for “python”.

Not installing pip only makes the setup faster, and it’s only about half a second for virtualenv (the default). That’s an absolute time, so great for simple environments, not as noticeable for big ones.

We have to pay the Python startup cost every Pip interaction due to no public API, also in build, etc. ;)

By the way, another speedup is uv doesn’t compile bytecode by default. For build, this makes sense, as you really just want to compile what you use and anything else is wasted. But not sure for nox. Most of the time, that probably is desired, but if you make a dev environment, it’s better to precompile. I’d want to measure the cost first, but it’s apparently enough that uv didn’t want to make it default.

[henryiii]

Yes, “venv-backend” also controls the installer. We didn’t have multiple installers before uv. Technically, you could mix and match, but it keeps the interface simpler. You can always install pip or uv in the other’s environment.

uv’s start up time is less than Python. In fact, it can finish making an empty venv before python -c “” can finish.

[henryiii]

Why twice?

[layday]

uv’s start up time is less than Python. In fact, it can finish making an empty venv before python -c “” can finish.

This difference is somewhere in the region of ~20 ms on my machine FWIW, which is outside the realm of the perceptible. python -m venv --without-pip completes in < 100 ms.

[layday]

Build 1.1 uses external pip if it can automatically

On the topic of build, readers might also be interested in https://github.com/pypa/build/pull/751, adding support for uv.

[pfmoore]

Yes, “venv-backend” also controls the installer. We didn’t have multiple installers before uv. Technically, you could mix and match, but it keeps the interface simpler. You can always install pip or uv in the other’s environment.

Fair enough. The documentation doesn't make this clear, which was the original point I was trying to make, and it suggests that the environment creation gains with uv are because it's simply faster, rather than actually being because it doesn't install pip. Claiming "uv environment creation is faster" is a sore point for me, as I specifically implemented the --python flag for pip so that we could start the (very slow!) process of eliminating the need to install pip everywhere. So having uv "steal" the credit for that speedup is something I probably get over-sensitive about (they get to ignore the backward compatibility issue, which is the only reason the pip/venv ecosystem is taking things so slowly...)

I think that mixing the two aspects might have been a mistake, and having a way to configure install backends (which could be "uv", "external pip" and "internal pip", plus whatever conda options apply) independently of environment creation options would have made it easier for the trade-offs to be presented. In practice, I imagine no-one will care that much about environment creation times, apart from eliminating the cost of installing pip (as @layday said, any remaining cost differences are imperceptible), but they will care about the time vs compatibility trade-offs with installation.

Anyway, as I said I'd be willing to help separate the venv-backend and install-backend behaviours, and make using an external pip more convenient, if that's something that people are interested in. But I don't want to force that on people if there's going to be a lot of pushback on the UI aspects.

[layday]

To add, I don't know how nox creates venvs, but I assume you'd be importing the venv module directly, as opposed to spawning a subprocess for uv, which is bound to be more costly than whatever venv does internally without pip.

[henryiii]

as I specifically implemented the --python flag for pip so that we could start the (very slow!) process of eliminating the need to install pip everywhere.

FYI, @pfmoore, https://fosstodon.org/@henryiii/112039400189486890 / https://twitter.com/HenrySchreiner3/status/1764755669461148138 is entirely from the --python flag for pip. :)

[henryiii]

Nox uses virtualenv by default, though there is an option to use venv. And in both cases, it actually calls the command rather than using the API, so there's some savings possible there.

[pfmoore]

Yeah, thanks - those savings in build are awesome 🙂

To add, I don't know how nox creates venvs, but I assume you'd be importing the venv module directly, as opposed to spawning a subprocess for uv, which is bound to be more costly than whatever venv does internally without pip.

That's an interesting point. I did a benchmark:

    def with_venv(target):
        builder = venv.EnvBuilder()
        builder.create(target)

    def with_uv(target):
        subprocess.run(["uv", "venv", "--quiet", target])

Averaged over 100 runs, venv took 0.007626 seconds, and uv took 0.072377 seconds. So yes, there's a significant speed benefit from using venv (without installing pip) in-process.

That's on Windows with Python 3.12.0. On Lunix (Ubuntu under WSL) venv was 0.001832 sec, uv was 0.564309 sec. I'm going to assume that's something to do with WSL, as otherwise that's a terrible result... Although a docker container with Python 3.12 is just as slow - I don't have a bare Linux machine to test on that.

And in both cases, it actually calls the command rather than using the API, so there's some savings possible there.

Ouch, yes there is. Running venv in a subprocess takes 15 times longer than running it in-process.

Here's my benchmarking script, for what it's worth: https://gist.github.com/pfmoore/a5406e0901609ed08cab8a0719d4e866