oliviagd
commented
9 months ago Open oliviagd opened 9 months ago
oliviagd
commented
9 months ago 
matteius
commented
9 months ago Does the Pipfile specify the custom index, because the pipenv install command examples you supply aren't passing the --index arg.
oliviagd
commented
9 months ago @matteius I do pass the custom index url when I run pipenv install, like this:
pipenv install "my-private-library[ingestor]==8.2.4.dev3+g2e01559.d20231005" -i https://my-private-artifactory-url
under 'Steps to Replicate' above I forgot to include it. Basically the issue is that installation works fine with pip but not with pipenv, eg the first command below cannot find my private extra whereas the pip3 install can
pipenv install "my-private-library[ingestor]==8.2.4.dev3+g2e01559.d20231005" -i https://my-private-artifactory-url
pip3 install "my-private-library[ingestor]==8.2.4.dev3+g2e01559.d20231005" -i https://my-private-artifactory-urlI am not sure right off where the issue could be or how I could reproduce -- the latest versions are the best yet at installing extras, but this report is unique in the use of private indexes. Do the wheels published there have the extras outlined in the METADATA file?
TiansuYu
commented
7 months ago I am encoutering a similar issue as well: My private libA has a dependency on another private libB. Both pip and pipx can install this libA without any issue when I feed it with the index url to look for libA. But for pipenv, I have to explicitly list the libB in the [packages] section in pipenv, and tell libB = {version="*", index="internal"}, alongside the libA={version="*", index="internal"}.
So pipenv clearly have no idea how to resolve transitive dependencies in private repo.
Even though pipenv does not respect pip.conf (I can live with that), but there should be a way to specify a project wide extra-index-url source, where it can use to resolve all dependencies for. (Similar to the Poetry approch.) (BTW, I am not mentioning Poetry to feed the fued, but just pointing out there is already an solution on the market, I don't see why you cannot copy the solution over to make your user's life easier.)
matteius
commented
7 months ago We explicitly removed support for extra-index-url because it is a vector for package confusion attacks. The default index is used unless otherwise specified with an index. This is the concept of index restricted packages that was introduced a year and a half ago.
matteius
commented
7 months ago @oliviagd following up on this -- have you found resolution or is support still requested?
TiansuYu
commented
6 months ago @matteius The attack surface can be minimized (I would even say mitigated) if you put your private repo in front of the public one. I don't think this feature should be disabled because of this. The disability only exists in Pipenv in the entire eco-system. I would recommend you reconsider this decision.
matteius
commented
6 months ago The attack surface can be minimized (I would even say mitigated) if you put your private repo in front of the public one.
@TiansuYu I don't think that is true.
The short answer is: there is no prioritization and you probably should avoid using --extra-index-url entirely.
Packages are expected to be unique up to name and version, so two wheels with the same package name and version are treated as indistinguishable by pip. This is a deliberate feature of the package metadata, and not likely to change.
https://github.com/pypa/pip/issues/5045#issuecomment-369521345
TiansuYu
commented
6 months ago Thanks for sharing.
From the stackoverflow link:
Pip does not really prioritize one index over the other in theory. In practice, because of a coincidence in the way things are implemented in code, it might be that one is always checked first, but it is not a behavior you should rely on.
I think pip's use base is large enough to obey Hyrum's law. And a lot of users do depend on how pip actually resolves the priority of the given index urls. (Or some other means to manage dependencies from both public pypi and private repos.) But it seems that my knowledge of this has been outdated:
https://github.com/pypa/pip/issues/8606#issuecomment-788124257
Prior to pip 21, index-url would take precedence in the case of equivalent files available, and since pip 21, it seems that either additional-index-url takes precedence or this choice is arbitrary (see https://github.com/pypa/pip/issues/9610).
Following up on the entire issue discussion https://github.com/pypa/pip/issues/8606 it seems like this is an issue requires a lot of efforts to get it done correctly. (Even the Poetry folks admits their solution is not perfect and full of shortcomings.)
I respect Pipenv's opinionated approach on this. But I do hope that Pipenv could provide a better UX in the end.
Issue description
Our team publishes our python libraries as .whl files to our private artifactory repository. I am trying to package one of our libraries with
extras. Some of the extras include dependencies from our private repository. When I try to pipenv install my package with extras from a private repository, pipenv seems to only be looking in pypi, even though the whl metadata accurately specifies the private repository as the source of the dependency.Expected result
My setup looks like this:
Installing extras that only come from pypi works fine:
Actual result
Installing the
ingestorextras includes an extra I'm callingmy-private-dependencythat is hosted on our artifactory. Theres a package on PyPi with an equivalent name that only has version 0.1 released as you can see in the error belowIf I start up a normal venv and install using pip it works fine
Additionally, the metadata of the whl file looks correct
Steps to replicate
I'm not sure how much information I can give here since I am working with proprietary artifacts. But in summary,