astral-sh / uv

An extremely fast Python package and project manager, written in Rust.
https://docs.astral.sh/uv
Apache License 2.0
27.51k stars 792 forks source link

Use users pip.conf #1404

Open msharp9 opened 9 months ago

msharp9 commented 9 months ago

Pip allows a user to set configuration settings inside a pip.conf, usually stored at ~/.config/pip/pip.conf. For example, setting a default index-url. uv seems to ignore these settings completely.

This is a problem since enterprise companies typically have their own PyPI images and don't want users to use the public PyPI for security reasons. For example, users who use AWS codeartifact have this config file updated automatically when they run aws login. https://docs.aws.amazon.com/codeartifact/latest/ug/python-configure-pip.html

zanieb commented 9 months ago

Thanks for the clear issue!

owenlamont commented 9 months ago

I was just about to raise an issue for this exact use case. We also use codeartifact for a few in house packages but mostly pull public PyPi packages via AWS codeartifact.

Anyway +1 to this. Would love to use uv if this can be supported.

kkpattern commented 9 months ago

We also set index-url in our venv by pip config --site set install.index-url "http://path/to/internal/pypi/server"

schlamar commented 9 months ago

BTW, for Windows this is %APPDATA%\pip\pip.ini, see https://pip.pypa.io/en/stable/topics/configuration/#location

mcrumiller commented 9 months ago

Can you perhaps add to the README instructions on how to set the index-url? My company uses a proxy repo and it's not obvoius how to read from it. Do we have to supply --index-url every time we call python -m pip uv? That seems a bit cumbersome.

zanieb commented 9 months ago

Hi @mcrumiller you can use UV_INDEX_URL instead. We may want to add all of our environment variables to the README, but they are currently listed in the --help output e.g.

  -i, --index-url <INDEX_URL>
          The URL of the Python package index (by default: https://pypi.org/simple)

          [env: UV_INDEX_URL=]
PetterS commented 8 months ago

The environment variables are nice, but reading the pip config file would be very good for enterprises. It would allow a seamless transition to uv without installing additional configuration on the machines.

edwardpeek-crown-public commented 8 months ago

We'd like the system config to be considered as well (/etc/xdg/pip/pip.conf on Ubuntu). Our organisation configure a local mirror for all dev machines & users this way.

RichardDally commented 8 months ago

The environment variables are nice, but reading the pip config file would be very good for enterprises. It would allow a seamless transition to uv without installing additional configuration on the machines.

Exactly, this is what I was expecting from uv, a drop-in replacement with a major performance boost.

zanieb commented 8 months ago

This is still under consideration, but the comment at https://github.com/astral-sh/uv/issues/1789#issuecomment-1978810917 explains our current stance well.

RichardDally commented 8 months ago

Hi @mcrumiller you can use UV_INDEX_URL instead. We may want to add all of our environment variables to the README, but they are currently listed in the --help output e.g.


  -i, --index-url <INDEX_URL>

          The URL of the Python package index (by default: https://pypi.org/simple)

          [env: UV_INDEX_URL=]

I've been trying environment variables but it raises an error related to Unexpected fragment on URL: contact. I've seen issues that have been closed for this, I'm a little bit puzzled here. Using version 0.1.12 on Windows on private Artifactory repository.

zanieb commented 8 months ago

@RichardDally could you please create a new issue if this is not related to pip.conf support?

zanieb commented 8 months ago

Okay, we are officially considering support for this feature. Here is my proposal.

Require explicit selection of configuration files

pip supports discovery of configuration files at various locations

Only support index configuration

pip supports all of the command-line options via the configuration file

Do not support configuration per command

pip supports separate configuration for each sub-command like install

kkpattern commented 8 months ago

Require explicit selection of configuration files

pip supports discovery of configuration files at various locations

  • Do not support automated discovery of pip.conf
  • Do not support global / user / site
  • Use of pip.conf requires opt-in e.g. --pip-config <path> or UV_PIP_CONFIG=<path>

Maybe UV can auto discovery pip.conf in a virtual environment?

UV_INDEX_URL is actually enough for most use cases. I think many want uv to support pip.conf because they want uv as a drop-in replacement for pip. Without auto-discovery pip.conf from a virtual env, this cannot be achieved.

Also, in our project's CI job, there're multiple stage install different dependencies into the same .venv. For example, the bootstrap script may setup the .venv and install all the essential dependencies. Then in a later stage, the test script will install all the test related dependencies into the .venv.

By auto-picking up the pip.conf in the virtual environment, all the jobs in the following stages don't need to worry about which index it should use, which is very convenient for us.

zanieb commented 8 months ago

Maybe UV can auto discovery pip.conf in a virtual environment?

What exactly do this mean? There's a pip.conf file in the .venv folder?

kkpattern commented 8 months ago

What exactly do this mean? There's a pip.conf file in the .venv folder?

Yes. If you activate a .venv then run pip config --site set install.index-url "http://pypi.internal.com", there will be a pip.conf in the .venv containing following content:

[install]
index-url = http://pypi.internal.com

Later pip commands will automatically pick up this config file.

zanieb commented 8 months ago

Why not just export an environment variable selecting the configuration file? e.g. UV_PIP_CONFIG=./.venv/pip.conf?

kkpattern commented 8 months ago

It's convenient when we want to install some libs into a .venv across multiple sessions.

For example, a developer may set up a local .venv for development and use it for a relatively long time. Then, one day, they may need to update or install a new lib from an internal Pypi index. Since pip can automatically pick up the pip.conf file in the .venvfolder, they don't need to remember what the index is or where the pip.conf is located.

This is not a deal breaker. We already use uv in our project and are very happy with it. But I think it's a nice-to-have feature.

zanieb commented 8 months ago

Thanks for the additional details! I don't think we'd launch this feature with support for that, but I'm not fully opposed. I worry about sometimes supporting inference, but having it work in the most-scoped context makes some sense.

notatallshaw commented 8 months ago

Just as another viewpoint for this conversation, as someone who has helped set up a lot of different users Python environments a lot of users get confused by all the different places pip configuration can go, e.g. environmental variables, multiple user directories, multiple system directories, venv directories etc.

I've encountered many time someone not being able to find where a setting was set, and have a broken environment until someone assisted them.

ikamensh commented 7 months ago

Do not support automated discovery of pip.conf Do not support global / user / site

Maybe one default location can be supported without causing any confusion? I would pick user config as the one config if I had to choose. I see how not picking any discovery is a simple solution, but perhaps there is a benefit if this choice is done well.

With environment variables the issue is e.g. I use fish shell, but sometimes things are run through bash. Then I have to maintain two configs and export variables there. Even for one shell its an extra step that sometimes could be avoided.

zanieb commented 7 months ago

The intent is to introduce our own configuration in the future and we'll support automatic discovery of that. We're just not particularly comfortable reading other tool's configuration files without opt-in.

msharp9 commented 7 months ago

Since I first created the issue there's been a lot of conversation, this is what I would suggest as I've thought about everyone's different viewpoints.

UV should create it's own configuration file and provide automatic discovery for it. There should then be another tool that converts pip configs to uv configs, e.g pip.conf -> uv.conf. This second tool could provide a simple extensible configuration to add default locations. Collect all pip.conf, and create one combined uv.conf to the auto discovery location.

If UV then added support for plugins, then all a user would have to do is pip install both, say, pip install uv uv-pip-conf. This would make the user experience feel much more like UV is actually a drop in replacement to pip and pip-tools since no flags or environment variables would have to be set.

This would be similar to how flake8 plugins work, simply pip install flake8 and flake8-import-order as an example, and now flake8 automatically checks your codes imports along with other syntax.

d-miketa commented 7 months ago

@zanieb Slightly off topic, but would you be willing to share an expected timeline for the config file feature? It’s the one thing preventing me from immediately swapping all of my projects to uv. 😅 Many thanks, love the project.

zanieb commented 7 months ago

Hey @d-miketa — we don't have a timeline yet. I think it will be relatively straight-forward to implement but I'm giving time for feedback on my proposal which only has four upvotes compared to the ~50 on the original post.

gaby commented 7 months ago

As stated by @PetterS not having built-in support for /etc/pip.conf will be detrimental for Enterprise adoption.

In our case we drop a configuration in /etc/pip.conf using Ansible, we then symlink this file to /etc/xdg/pip/pip.conf for Ubuntu based systems.

After that any Python package installed using ansible.pip module automatically uses the configuration from /etc/pip.conf, we don't have to expose ENV's or write extra config files.

charliermarsh commented 7 months ago

That requirement sounds more like you need a persistent configuration file rather than a pip.conf specifically - or am I misunderstanding?

gaby commented 7 months ago

@charliermarsh In enterprise environments /etc/pip.conf is used to managed several things:

User are just able to do "pip install" without having to configure anything since it's managed by the platform via /etc/pip.conf.

astronautas commented 7 months ago

Any progress on this? It's critical for enterprise environments.

CostcoFanboy commented 7 months ago

The intent is to introduce our own configuration in the future and we'll support automatic discovery of that. We're just not particularly comfortable reading other tool's configuration files without opt-in

Then uv will stay indefinitely a non-viable product for enterprise solutions.

/etc/pip.conf or even sometimes pip.conf files getting cp'd inside of the venv folder during the dependency installation phase is an industry-wide practice/trend, for better or for worse.

Regardless, huge fan of the product and thank you immensely for your time and hard work. Just sad to see this will only stay on personal projects.

charliermarsh commented 7 months ago

I think what I don't understand yet is: if you're going through the effort to migrate from pip to uv, why is it any harder to ask that you use (for sake of argument) uv.conf rather than pip.conf? Why is that not possible? Like, if I told you that it was the exact same file, but it had to be named uv.conf, why would that be a problem?

We may be convinced to change the behavior here (I'm genuinely looking for answers to that question), but FWIW I can personally name a bunch of enterprises that are already using uv, so I'm not worried about it only being viable for personal projects :)

gaby commented 7 months ago

@charliermarsh If uv.conf becomes a thing, and location is predefined, ex /etc/uv.conf. It should work enterprise environments.

charliermarsh commented 7 months ago

That’s the thing I’m mostly trying to understand. Is it important the file is literally named pip.conf, that it follows the exact same schema, that it’s auto-discovered in the same way? If so, why? (I’m genuinely open to being convinced, it just adds work and a maintenance burden for us so I want to make sure we have the right motivation.)

Or is it that users just need a way to specify persistent configuration? What if we (eg) supported reading pip.conf but you had to set a UV environment variable for it?

CostcoFanboy commented 7 months ago

I think what I don't understand yet is: if you're going through the effort to migrate from pip to uv, why is it any harder to ask that you use (for sake of argument) uv.conf rather than pip.conf? Why is that not possible? Like, if I told you that it was the exact same file, but it had to be named uv.conf, why would that be a problem?

Apologies I don't think I expressed myself clearly.

While the actual name of the file doesn't necessarily need to be the same, the feature set need to be mirrored and it would be highly desirable for the discovery mechanism to be mimicked. Mostly having a global predefined location (e.g. /etc/pip.conf) and a project-specific/localized one (e.g. inside of venv in its root) is very important..

From the previous posts it didn't feel like there was functionality-mirroring in mind, but from your clarification it does feel like I'm wrong.

FWIW I can personally name a bunch of enterprises that are already using uv, so I'm not worried about it only being viable for personal projects :)

Would this mean there is another way that UV supports trusted-host, index-url AND extra-index-url (fallback)?

hutch3232 commented 7 months ago

In my enterprise environment, sys admins configure /etc/pip.conf to work with our PyPI mirror. A sizeable portion of our users probably do not really even understand the mirror or bother with pip.conf since it "just works". Thinking short-term, it would be nice if uv picked up pip.conf so that it was a drop-in replacement for pip in enterprise environments. It wouldn't require convincing the sys admins to create/maintain an additional config file or to teach users how to do it themselves. Longer-term though, as uv adoption increases, it probably wouldn't be much of an ask for sys admins to create the file. I think supporting pip.conf would primarily help less comfortable users get up and running with uv. More advanced users would have little issue figuring it out, using a hypothetical uv.conf or environmental variables. With that in mind, less comfortable users probably are sticking with pip in the short-term anyway. More advanced users are probably the people very excited about uv and jumping into it.

astronautas commented 7 months ago

@charliermarsh In our case (a small data point, but still) we don't care that much how the file is named, as long as uv can resolve deps from non pypi sources and it's possible to define those sources within *.conf.

PetterS commented 7 months ago

There can be (and have been where I've worked) automated setup scripts that make sure there is a pip.conf with appropriate settings. If uv just works with that file the gradual adoption within the enterprise could be quicker.

It's not a dealbreaker of course since we are talking about developers here.

eswolinsky3241 commented 6 months ago

I think what I don't understand yet is: if you're going through the effort to migrate from pip to uv, why is it any harder to ask that you use (for sake of argument) uv.conf rather than pip.conf? Why is that not possible? Like, if I told you that it was the exact same file, but it had to be named uv.conf, why would that be a problem?

We may be convinced to change the behavior here (I'm genuinely looking for answers to that question), but FWIW I can personally name a bunch of enterprises that are already using uv, so I'm not worried about it only being viable for personal projects :)

@charliermarsh Just to comment since I came across this post while playing with uv: We also use AWS Codeartifact, which uses short-lived credentials that need to be refreshed every twelve hours. The AWS CLI offers a single helper command that can obtain the credentials and rewrite the index url in pip.conf (or the equivalent config file for npm, maven, etc.). Obviously AWS currently offers no such support for uv, so it's on the user to update uv.conf every twelve hours. Not saying this is a huge deal, just something that could be tedious enough to convince an enterprise user to stick with pip. Loving uv so far though!

ilCatania commented 6 months ago

I also work in a corporate environment where someone else manages pip.conf on my behalf and I do not have full control over my users' installations.

My short term plan to be able to replace pip with uv pip has been a handmade utility that reads from pip.conf and sets the corresponding uv environment variables.

I would be very happy, once uv supports its own configuration file, if there was a command to create / update said configuration file with values from pip.conf. Something like

uv import-pip-conf --overwrite --ignore-unsupported --pip-conf-file=~/.pip.conf

(I am making it up but you get the idea hopefully)

quartox commented 1 month ago

I also have the short-lived credentials problem in an corporate environment. My pip.conf file is updated every 8 hours with new credentials. The tools to update these credentials are owned by other teams at my company. It would be more challenging for me to convince them to adopt uv before I am able to test it out and demonstrate some improvements with existing workflows.

I was wondering if it would it be possible for uv.conf to reference specific configurations in pip.conf? For example what if the uv.conf file contains the line

index-url = ${/path/to/pip.conf:index-url}

Don't care much about specific syntax (which I took from here) but the intent is to reference values in the pip.conf file explicitly. This way we could keep updating pip.conf every 8 hours until I can get broad consensus to move to uv.conf workflows.

T-256 commented 1 month ago

FYI, https://github.com/astral-sh/uv/pull/7851 landed, which now allows you to set global configuration for uv.toml.

darksim33 commented 1 month ago

FYI, #7851 landed, which now allows you to set global configuration for uv.toml.

Just to make this clear since I'm new to the tool. Is it possible to link the pip.conf using the uv.toml? I tried a few things but they don't seem to work.

gaby commented 1 month ago

@charliermarsh In enterprise environments /etc/pip.conf is used to managed several things:

  • Index-url
  • Search-index
  • HTTP/HTTPS proxies
  • Custom CA's

User are just able to do "pip install" without having to configure anything since it's managed by the platform via /etc/pip.conf.

Quoting myself since I just tried using /etc/uv/uv.toml and some of these options are not even supported.

uv keeps pushing some of these configs to ENV variables. These are not practical for enteprise environments.

Currently there's no way to set a cert file for a proxy/index, and there's no way to set http/https proxies via pyproject.toml / uv.toml

Enteprise environments typically set SquidProxy or Pypi Mirror. These services are exposed to users with internal TLS certs signed by the Enterprise CA.

tldr: If the ENV's listed here https://docs.astral.sh/uv/configuration/environment/ were supported via uv.toml and pyproject.toml it would solve a lot of the use-cases listed on this thread.

schlamar commented 1 month ago

@gaby for this use case you can use native-tls setting https://docs.astral.sh/uv/reference/settings/#native-tls

In this case SSL works automatically if your internal cert is in the system's certificate store (which is usually the case in enterprise setups).

gaby commented 1 month ago

@schlamar Yes, that works. But sometimes the Proxy CA is not installed in the system.

Regardless, you can't set http/https proxy via toml only via ENV.