Custom sources for python distribution binaries

[BrendanJM]

Many companies have policies around sourcing artifacts from internal sources only. I'm seeing in the docs that uv uses pre-built third-party distributions from the python-build-standalone project. Is it possible to configure uv to point to an alternate private index for python version distributions?

I imagine as long as there's a well-defined API for how uv queries and returns binaries, the actual source from which these are downloaded should probably not matter much?

Apologies if this is a duplicate, I searched around but didn't spot an existing issue.

[Crypto-Spartan]

Timing is wild, I was just about to submit an issue with this same request. I would love support for changing the URL that uv uses for it's request to the python-build-standalone project as well. (also due to a corporate environment)

[zanieb]

Yep! UV_PYTHON_INSTAL_MIRROR

[zanieb]

Described at https://docs.astral.sh/uv/configuration/environment/

[Crypto-Spartan]

Thank you for the quick response!

Additionally, is there support for providing uv a filepath instead of a URL to the .tar.gz that's downloaded from the python-build-standalone project (cpython-3.X.X+<date>-<target>-install_only_stripped.tar.gz)?

[zanieb]

You can use file:// in the install mirror.

[Crypto-Spartan]

so I'm currently trying to install https://github.com/indygreg/python-build-standalone/releases/download/20241008/cpython-3.12.7+20241008-x86_64_v3-unknown-linux-gnu-pgo+lto-full.tar.zst with uv.

I have it downloaded in $HOME/downloads/python_standalone/20241008/cpython-3.12.7+20241008-x86_64_v3-unknown-linux-gnu-pgo+lto-full.tar.zst

I set UV_PYTHON_INSTALL_MIRROR='file:///home/<user>/downloads/python_standalone'

When I attempt to run uv python install 3.12.7, it fails with

Searching for Python versions matching: Python 3.12.7
error: No download found for request: cpython-3.12.7-linux-x86_64-gnu

> echo $UV_PYTHON_INSTALL_MIRROR
file:///home/<user>/downloads/python_standalone

Could you provide help/guidance as to how I should proceed? Thanks for your help

I also tried with UV_PYTHON_INSTALL_MIRROR='file://home/<user>/downloads/python_standalone' just incase I had too many slashes after the file:, still didn't work

I'm on Ubuntu 22.04 LTS (jammy) if that matters

[zanieb]

Can you share verbose logs? i.e. RUST_LOG=uv=trace uv python install -v 3.12.7

Your uv version needs to support that managed download, so you might just be on a uv version that only has metadata for 3.12.6 and not 3.12.7 yet?

[Crypto-Spartan]

> RUST_LOG=uv-trace uv python install -v 3.12.7
DEBUG uv 0.4.18
TRACE Checking lock for `/home/<user>/.local/share/uv/python` at `/home/<user>/.local/share/uv/python/.lock`
DEBUG Acquired lock for `/home/<user>/.local/share/uv/python`
Searching for Python versions matching: Python 3.12.7
TRACE ld path: /lib64/ld-linux-x86-64.so.2
TRACE stdout output from `ld`: ""
TRACE stderr output from `ld`: "/lib64/ld-linux-x86-64.so.2: missing program name\nTry '/lib64/ld-linux-x86-64.so.2 --help for more information.\n"
TRACE stdout ouput from `ldd --version`: ld.so (Ubuntu GLIBC 2.35-0ubuntu3.8) stable release version 2.35.\nCopyright ...
TRACE Found manylinux 2.35 in stdout of `ldd --version`
DEBUG Released lock at `/home/<user>/.local/share/uv/python/.lock`
error: No download found for request: cpython-3.12.7-linux-x86_64-gnu

uv 0.4.18 is the latest version of uv I'm able to grab off of the pypi mirror in my corporate environment

[zanieb]

3.12.7 was added in https://github.com/astral-sh/uv/pull/7880 which is in uv 0.4.19

[Crypto-Spartan]

Tried again with https://github.com/indygreg/python-build-standalone/releases/download/20240909/cpython-3.12.6+20240909-x86_64-unknown-linux-gnu-pgo+lto-full.tar.zst instead this time.

> uv python install 3.12.6
Searching for Python versions matching: Python 3.12.6
cpython-3.12.6-linux-x86_64-gnu
error: Failed to install cpython-3.12.6-linux-x86_64-gnu
  Caused by: Failed to extract archive: cpython-3.12.6+20240909-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz
  Caused by: Invalid gzip header
cpython-3.12.6-linux-x86_64-gnu

[Crypto-Spartan]

does uv only install the x86_64-unknown-linux-gnu-install_only_stripped.tar.gz? Why can't I install the x86_64-unknown-linux-gnu-pgo+lto-full.tar.zst?

[zanieb]

Sorry this isn't well documented — yes, we only support the target defined in our download metadata.

We could probably support more in the future, it's just more complicated.

[zanieb]

Basically the reason you can't, is that we generate JSON metadata from GitHub release

https://github.com/astral-sh/uv/blob/f6fd849f2c63e3b319ab03c936beab866dc303ba/crates/uv-python/download-metadata.json#L398-L409

then encode it as static Rust code for performance

https://github.com/astral-sh/uv/blob/f6fd849f2c63e3b319ab03c936beab866dc303ba/crates/uv-python/src/downloads.inc#L471-L484

then when you ask for a custom mirror we transform the static URL

https://github.com/astral-sh/uv/blob/891c91dd3a0141f05adea20674f3ea7f4477c510/crates/uv-python/src/downloads.rs#L577-L585

This is quite a naive implementation, but it's intended to unblock core functionality not be fully feature complete.

[Crypto-Spartan]

Is there any specific reason why the generated json only includes the install_only_stripped.tar.gz and none of the other python standalone builds? I'm willing to work on a PR to support the other builds if you're open to accepting it.

[zanieb]

I think we just adopted the "choose a single build" behavior from Rye's code. I'd love to support other builds (see #8019), though it's probably non-trivial. If you want to work on it, I'd appreciate it. Otherwise I'll probably work on #8019 in the near future.

[Crypto-Spartan]

something weird is happening still, it looks like uv is attempting to download from github directly still which won't be possible in my environment.

cpython-3.12.6+20240909-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz is currently in /home/<user>/downloads/python_standalone

> echo $UV_PYTHON_INSTALL_MIRROR
file:///home/<user>/downloads/python_standalone

> RUST_LOG=uv-trace uv python install -v 3.12.6
DEBUG uv 0.4.18
TRACE Checking lock for `/home/<user>/.local/share/uv/python` at `/home/<user>/.local/share/uv/python/.lock`
DEBUG Acquired lock for `/home/<user>/.local/share/uv/python`
Searching for Python versions matching: Python 3.12.6
TRACE ld path: /lib64/ld-linux-x86-64.so.2
TRACE stdout output from `ld`: ""
TRACE stderr output from `ld`: "/lib64/ld-linux-x86-64.so.2: missing program name\nTry '/lib64/ld-linux-x86-64.so.2 --help for more information.\n"
TRACE Tried to find musl version by running `"/lib64/ld-linux-x86-64.so.2"`, but failed: Could not find musl version in output of: `/lib64/ld-linux-x86-64.so.2`
TRACE Tried to find libc version from possible symlink at "/lib64/ld-linux-x86-64.so.2", but failed: Failed to find glibc version in the filename of linker: `/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2`
TRACE stdout ouput from `ldd --version`: ld.so (Ubuntu GLIBC 2.35-0ubuntu3.8) stable release version 2.35.\nCopyright ...
TRACE Found manylinux 2.35 in stdout of `ldd --version`
DEBUG Using request timeout of 30s
TRACE Handling request for https://github.com/indygreg/python-build-standalone/releases/download/20240909/cpython-3.12.6+20240909-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz
TRACE Request for https://github.com/indygreg/python-build-standalone/releases/download/20240909/cpython-3.12.6+20240909-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz is unauthenticated, checking cache
TRACE No credentials in cache for URL https://github.com/indygreg/python-build-standalone/releases/download/20240909/cpython-3.12.6+20240909-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz
TRACE Attempting unauthenticated request for https://github.com/indygreg/python-build-standalone/releases/download/20240909/cpython-3.12.6+20240909-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz
DEBUG Downloading https://github.com/indygreg/python-build-standalone/releases/download/20240909/cpython-3.12.6+20240909-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz to temporary location: /home/<user>/.local/share/uv/python/.cache/.tmpYs2Aii
DEBUG Extracting cpython-3.12.6%2B20240909-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz
error: Failed to install cpython-3.12.6-linux-x86_64-gnu
  Caused by: Failed to extract archive: cpython-3.12.6+20240909-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz
  Caused by: Invalid gzip header
DEBUG Released lock at `/home/<user>/.local/share/uv/python/.lock`

[zanieb]

On my machine, I get something like this if I specify an arbitrary directory:

❯ UV_PYTHON_INSTALL_MIRROR="file:///Users/zb/test" RUST_LOG=uv=trace uv python install -v 3.12.6
DEBUG uv 0.4.18
TRACE Checking lock for `/Users/zb/.local/share/uv/python` at `/Users/zb/.local/share/uv/python/.lock`
DEBUG Acquired lock for `/Users/zb/.local/share/uv/python`
Searching for Python versions matching: Python 3.12.6
DEBUG Using request timeout of 30s
error: Failed to install cpython-3.12.6-macos-aarch64-none
  Caused by: failed to query metadata of file `/Users/zb/test/20240909/cpython-3.12.6+20240909-aarch64-apple-darwin-install_only_stripped.tar.gz`
  Caused by: No such file or directory (os error 2)
DEBUG Released lock at `/Users/zb/.local/share/uv/python/.lock`

[zanieb]

Similarly:

❯ wget https://github.com/indygreg/python-build-standalone/releases/download/20240909/cpython-3.12.6%2B20240909-aarch64-apple-darwin-install_only_stripped.tar.gz
❯ mkdir 20240909
❯ mv cpython-3.12.6+20240909-aarch64-apple-darwin-install_only_stripped.tar.gz 20240909
❯ UV_PYTHON_INSTALL_MIRROR="file:///Users/zb/workspace/uv" RUST_LOG=uv=trace uv python install -v 3.12.6
DEBUG uv 0.4.18
TRACE Checking lock for `/Users/zb/.local/share/uv/python` at `/Users/zb/.local/share/uv/python/.lock`
DEBUG Acquired lock for `/Users/zb/.local/share/uv/python`
Searching for Python versions matching: Python 3.12.6
DEBUG Using request timeout of 30s
DEBUG Downloading file:///Users/zb/workspace/uv/20240909/cpython-3.12.6%2B20240909-aarch64-apple-darwin-install_only_stripped.tar.gz to temporary location: /Users/zb/.local/share/uv/python/.cache/.tmpvg3LcW
DEBUG Extracting cpython-3.12.6%2B20240909-aarch64-apple-darwin-install_only_stripped.tar.gz
DEBUG Moving /Users/zb/.local/share/uv/python/.cache/.tmpvg3LcW/python to /Users/zb/.local/share/uv/python/cpython-3.12.6-macos-aarch64-none
Installed Python 3.12.6 in 380ms
 + cpython-3.12.6-macos-aarch64-none
DEBUG Released lock at `/Users/zb/.local/share/uv/python/.lock`

[Crypto-Spartan]

Figured it out

my filename was urlencoded so was actually cpython-3.12.6%2B20240909-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz instead of cpython-3.12.6+20240909-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz (%2B vs +)

Thanks for all of your help

[zanieb]

That makes sense. We can add more documentation around this.

[guillemc23]

I'm a bit confused. We're currently migrating to uv and we need to install python packages from private sources.

Currently, I'm just adding the following to my pyproject.toml:

[tool.uv]
index-url = "https://<user>:<password>@<host>"

Which is working properly.

I have been unable to find a way to configure this authentication in order to avoid just hardcoding the credentials for obvious reasons. Is this somehow related to this issue? Also, is this the way I should be doing this?

EDIT: I believe what I'm referring to is this PR https://github.com/astral-sh/uv/pull/7741

[chrisrodrigue]

Just took a look at #7741 and I am wondering why the environment variables can't be specified in the pyproject.toml for the index URLs.

[zanieb]

@guillemc23 sorry this issue is about downloading Python distributions — not index URLs for packages. Could you open a new issue for your question instead?

[MeitarR]

Figured it out

my filename was urlencoded so was actually cpython-3.12.6%2B20240909-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz instead of cpython-3.12.6+20240909-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz (%2B vs +)

Thanks for all of your help

Opened PR with a create mirror script to make it easier https://github.com/astral-sh/uv/pull/8548