conda / conda-lock

Lightweight lockfile for conda environments
https://conda.github.io/conda-lock/
Other
458 stars 102 forks source link

Improve auth configuration for private PyPi repositories #460

Open jacksmith15 opened 11 months ago

jacksmith15 commented 11 months ago

Checklist

What is the idea?

I'd like to improve upon the mechanism for configuring auth for private PyPi repositories.

Why is this needed?

The documentation claims the process for configuring private pypi repositories is simply:

poetry config repositories.foo https://username:password@foo.repo/simple/

However I believe the process actually looks as follows:

  1. Install the correct version of poetry (matching the vendored version):
    pipx install poetry==1.1.15
  2. Configure repositories using poetry:
    poetry config repositories.foo https://username:password@foo.repo/simple/ 
  3. Copy poetry config to location discoverable by conda-lock:
    # Note that exact paths vary on different machines
    cp -R ~/.config/pypoetry ~/.config/pypoetry-conda-lock

I think there are a few problems here:

  1. Documentation not up-to-date
  2. Users must install poetry separately, at a specific version matching the vendored poetry. This negates the value of vendoring in the first place.
  3. Users use an interface provided by poetry, which is effectively an implementation detail of conda-lock (currently conda-lock uses poetry as a resolver under the hood, but conda-lock should be free to change the resolution algorithm without breaking user workflows).
  4. Authentication secrets are saved to and copied around on disk.

What should happen?

I propose that conda-lock provides its own interface for configuring private pypi repositories, and then manages the internal gymnastics of providing these to the poetry resolver.

I propose that this interface supports configuration via environment variable first-and-foremost, as this is the most portable approach for configuration, and supports more secure workflows with tools like envchain.

Additional Context

Note also separate issue https://github.com/conda/conda-lock/issues/461 regarding auth stripping for private PyPi repositories.

maresb commented 11 months ago

Thanks a lot @jacksmith15 for your thoughts on this and #461. I haven't read in much detail, but the situation here is indeed quite messy. I'm going to take a crack right now at upgrading the vendored Poetry. This will hopefully improve the baseline somewhat. Ultimately I think we shouldn't be relying on Poetry, but I'm just going to focus on the upgrade for now...

In case this seems to have dropped off my radar don't hesitate to ping me.