Add/improve support for Path dependencies and GIT dependencies

romain-intel commented 2 months ago

Specifically:

adds subdirectory for GIT dependencies
adds the ability to pass "path" dependencies to Poetry

Description

This simply extends support of the arguments that conda-lock can parse in a pyproject.toml and passes them down to Poetry. Specifically it adds support for local directories and wheels as well as support for #subdirectory in GIT packages.

netlify[bot] commented 2 months ago

Deploy Preview for conda-lock ready!

Name	Link
Latest commit	c65d095d8a3d131523dd5a75fc94d18bfe1f4959
Latest deploy log	https://app.netlify.com/sites/conda-lock/deploys/66fee7a2aeb8260008d3b0d2
Deploy Preview	https://deploy-preview-712--conda-lock.netlify.app
Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

romain-intel commented 2 months ago

Need to add a test for subdirectory.

romain-intel commented 2 months ago

I suspect the micromamba test failures may be due to the release of micromamba 2.0

maresb commented 2 months ago

I suspect the micromamba test failures may be due to the release of micromamba 2.0

Noooooo! :scream: I wanted to release! Ok, I guess we have more work.

Regarding this PR, it looks mostly great. One of my big concerns is favoring Poetry-style pyproject.toml over standards-based pyproject.toml. I think/hope that this would be a really simple tweak to achieve this. I think you basically just need to support

parse_python_requirement("/path/to/some-package", manager="pip", mapping_url=DEFAULT_MAPPING_URL)

Looking forward to the additional test for subdir. Thanks again @romain-intel!!!

maresb commented 2 months ago

Ok, micromamba v2 was not so bad. I believe I took care of it in https://github.com/conda/conda-lock/pull/713. Feel free to rebase on main, and I hope that will fix things.

romain-intel commented 1 month ago

Hopefully that works -- the PEP for local requirements seems to be mypkg @ /path/to/mypkg so that is what I am testing here. Let me know if you need anything else.

maresb commented 1 month ago

On first glance this looks great, thanks so much @romain-intel!!!

Note one hiccup in the doctest with a missing subdir=None.

romain-intel commented 1 month ago

whoops. Sorry about that. Hopefully that will do it.

Unrelated: I see a flurry of activity, planning a release soon?

romain-intel commented 1 month ago

Gah -- something up with windows :( . I can't seem to retrigger tests but I think everything else should be fine :).

maresb commented 1 month ago

Oops, https://github.com/conda/conda-lock/pull/717 should fix it. Could you please rebase again?

maresb commented 1 month ago

Unrelated: I see a flurry of activity, planning a release soon?

Yes, I really want to get this released as soon as possible. There are a few tricky things though. The release of micromamba 2 is still causing a lot of chaos, e.g. https://github.com/conda/conda-lock/pull/709, so we should probably let that settle down a bit first.

romain-intel commented 1 month ago

Rebased.

And yes, micromamba v2 is also causing me headaches :). Hopefully you can release soon though. Definitely interested in getting this large change integrated.

maresb commented 1 month ago

I'm going to take all the green I can get and merge this. Feel free to do any followup in a subsequent PR.

romain-intel commented 1 month ago

Thanks -- yes, sorry, busy day. Thanks for merging!!

maresb commented 1 month ago

I had a concern regarding the parse_python_requirement implementation for path dependencies, but now I'm convinced it's not that big of a deal.

We're assuming a nice PEP 508 compliant input string where the name is specified. However, pip is unfortunately a bit more loose about actually specifying the name.

So if I do:

from conda_lock.lookup import DEFAULT_MAPPING_URL
from conda_lock.src_parser.pyproject_toml import parse_python_requirement

parse_python_requirement(
    "./downloads/numpy-1.9.2-cp34-none-win32.whl", mapping_url=DEFAULT_MAPPING_URL
)

then I get

InvalidRequirement: Expected package name at the start of dependency specifier
    ./downloads/numpy-1.9.2-cp34-none-win32.whl
    ^

That error message is a lot more informative than I was expecting, which is why I don't think this is so serious of a problem.

Since not everyone has recently read PEP 508, it might be kind to intercept this exception and reraise with a note explaining that names can be specified by package-name @ .... But I wouldn't block the release on this.

romain-intel commented 1 month ago

I can make a more informative message. I'll try this over the weekend or early next week. And yes, I am aware of that and it is spec compliant if not the easiest. I think pip extracts the name from the setup or something when not specified. So nice but then again, why have specs :).

conda / conda-lock