[WIP] support poetry sources for private pypi repositories

[croth1-liveeo]

While private sources are currently supported, they do not use the poetry source concept in pyproject.toml: While specifying source = pypi will mark a package to be installed with pip instead of conda, it does not allow specifiying which index the package should come from, nor allow configurig the used indices directly in pyproject.toml.

The changes in this PR should make conda-lock aware of poetry sources and also allow providing auth for private sources.

Relies on: https://github.com/conda/conda-lock/pull/323 for providing auth to private indices during installation.

Still WIP - tests still needed.

[netlify[bot]]

✅ Deploy Preview for conda-lock ready!

Name	Link
🔨 Latest commit	f09744dc8520345a046e8e4a1217656a07ca496b
🔍 Latest deploy log	https://app.netlify.com/sites/conda-lock/deploys/63f0da59b220980008f6f836
😎 Deploy Preview	https://deploy-preview-353--conda-lock.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[maresb]

Thanks a lot for all this work! I do however have one serious concern, namely that we are favoring Poetry over the alternatives which are more standards-compliant. For example, the source = "pypi" key-value pair is not standards-compliant, so unless I'm mistaken it can only really be used for projects using Poetry as a build system. I'd really like to see this somehow eliminated.

Unfortunately I don't yet have any particular system in mind. It could be that we continue leveraging certain vendored components of Poetry, like regarding authentication, while ensuring that the functionality is available to non-Poetry users. Do you have any thoughts?

[maresb]

I just saw that the CI tests were locked due to your new username. Feel free to ping me if you need CI approval.

To elaborate a bit on what I wrote above, I'd be interested to see if we could swap out our vendored Poetry with something more standards-compliant like PDM. They use project.dependencies, which would unfortunately leave basic auth credentials in pyproject.toml. But to get around this, they also support environment variable expansion.

Given this, my main concern is to ensure that new functionality we add doesn't lock us in further towards Poetry.

[croth1-liveeo]

I have to admit, I have very limited understanding of current python build systems - last time I checked, everybody was using setup.py and pip :grimacing:.

For me the main motivation here was to make conda-lock and poetry compatible, as for us conda usage is limited to data science folks and supporting both poetry and conda compatible pyproject.toml seems very advantageous.

But I agree that poetry is just one system among many and building a more general repository system for conda-lock would have a lot of charm. However, that would require some research and input. Last time I checked, there was no such thing as lock files and package installation order was specified in pip.conf. :grimacing:

[maresb]

Yes, the state of current Python build systems is quite a mess, although it is improving since the old setuptools days.

I'm happy to have Poetry-compatibility so long as it's not at the cost of compatibility with other tools.

Research and input is definitely welcome. My own understanding is quite limited (I haven't even used PDM), so I would definitely miss things without these sorts of discussions.

[croth1-liveeo]

Okay, having thought about this a bit more: logically, I think the concept of poetry sources and conda channels is completely equivalent, and a lot would become cleaner if instead of defining them separately here, we just use the concept of channel, which is a conda_channel in case of a conda dependency and the name of some kind of python pypi repository in case of a pip repository.

https://github.com/conda/conda-lock/pull/353/files#diff-c6519186a02b9314f4ed76075c3b00068f2a440442c6ee64c0690f44c82ce3e0R61-R62

We could let tools like poetry specify the repositories in pyproject.toml or we just go with the current way that repositories and auth are read out of some config file. At the moment a lot of the logic is a bit tied to poetry, because that's in the end the code that does the solving. But logically the concept of a generalized channel that provides the package should be the same.

[croth1-liveeo]

@maresb I refactored the code a bit and now use the channel concept both for conda channels and pypi repositories. I think if there's a need, we could relatively easily generalize adapt the code to support private indices for non-poetry pyproject.yaml.

One way would be by defining pypi repositories in a new section tool.conda-lock.pypi_repositories (similarly to poetry sources) and then passing the auth via command line e.g. If we wanted to also support defining a specific source repository for packages, we would need another tool.conda-lock section, that would provide this mapping.

[maresb]

Thanks @croth1-liveeo!

This last commit looks primarily like a substitution poetry → pypi, which actually does make me feel a bit better. Am I understanding correctly that the channel concept would be future work?

I'll try to look more carefully soon.

[croth1-liveeo]

If there's interest, I could build support also for the non-poetry-flavored pyproject.toml (albeit not on company time :grimacing:) This PR should move conda-lock a bit closer to fully supporting poetry-style pyproject.toml, while being general enough to not introduce unnecessary poetry internals in the process. I believe the PR should be fully backwards compatible - it just extends the functionality for poetry pyproject.toml. I personally would prefer to extend on this in a separate PR.

I will write tests, once you had the chance to go through this PR and have no concerns on a conceptual level :slightly_smiling_face:

[leeleavitt]

What info could i provide you during my lock that might help troubleshoot why i'm unable to lock?

[maresb]

I was deep in the debugger for about 2 hours

@leeleavitt, it's obviously not ideal that a debugger is necessary to understand what the code is doing. Any annotations you could contribute in the form of code comments would be very welcome.

[maresb]

Please correct me if I'm wrong, but I think the desired functionality is now provided by #529