Can't install packages from private static repository

lecardozo commented 2 years ago

[x] I am on the latest Poetry version.
[x] I have searched the issues of this repo and believe that this is not a duplicate.
[x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

OS version and name: macOS Monterey (12.2.1)
Poetry version: 1.1.13
Link of a Gist with the contents of your pyproject.toml file: pyproject.toml

Issue

Hey, everyone! :D

I'm trying to use Poetry as the dependency manager for one of our internal applications. Here we have an internal private PyPI repository that is essentially a static index (served on Github Pages) following the https://<DOMAIN>/python-package-server/<PACKAGE_NAME>/ URL pattern. For each package, we have a list of links to specific repositories where the artifacts live in. Here is an example of what a page looks like for a package in this repository. 👇

<!DOCTYPE html>
<html>
  <head>
      <title>Links for package</title>
  </head>
  <body>
    <h1>Links for package</h1>
    <!-- package-server-links-start -->
    <a href="git+https://github.com/<redacted>/package@2.0.0#egg=package-2.0.0">package-2.0.0</a><br/>
    <a href="git+https://github.com/<redacted>/package@1.1.0#egg=package-1.1.0">package-1.1.0</a><br/>
    <a href="git+https://github.com/<redacted>/package@1.0.0#egg=package-1.0.0">package-1.0.0</a><br/>
  </body>
</html>

After adding the private source to pyproject.toml, the resolver was unable to find matching versions for that package

Loading configuration file /Users/me/Library/Application Support/pypoetry/config.toml
Loading configuration file /Users/me/projects/my-project/poetry.toml
Adding repository private (https://<DOMAIN>/python-package-server)
Using virtualenv: /Users/me/projects/my-project/.venv
[urllib3.connectionpool] Starting new HTTPS connection (1): <DOMAIN>:443
[urllib3.connectionpool] https://<DOMAIN>:443 "GET /python-package-server/package/ HTTP/1.1" 304 0
private: 0 packages found for package * # <---- at least it tried :) 
private: 0 packages found for package * # <---- once again? maybe 
[urllib3.connectionpool] Starting new HTTPS connection (1): pypi.org:443
[urllib3.connectionpool] https://pypi.org:443 "GET /pypi/package/json/ HTTP/1.1" 404 1988
PyPI: No packages found for package *

I guess this is happening because this format doesn't seem to be fully PEP 503 complaint, although we never had issues with PIP or PDM before.

After checking for previous discussions, I ended up finding this PR which seems to be related but I'm not completely sure. I've tested that specific version (from 5517) to see if it would work on my use case, but, unfortunately, it didn't.

Any clues on what could be the issue here? 🤔

Thanks!

abn commented 2 years ago

Hi @lecardozo #5517 is aimed at single page PEP 503 repositories only. This won't work as it is referring to VCS sources. The pip and PDM use cases for these sources are different to that of Poetry. You might have better luck either pointing them to release artifacts - ie. valid sdist fiels or wheels instead of a git repository.

lecardozo commented 2 years ago

Thanks for the quick reply @abn ! 😃

The pip and PDM use cases for these sources are different to that of Poetry. You might have better luck either pointing them to release artifacts - ie. valid sdist fiels or wheels instead of a git repository.

I see. So the limitation is in the fact that Poetry can't handle links to a specific tag in a repository (to be cloned and installed), only direct links to artifacts (sdists or wheels). Is that correct? If so, is it something that poetry might support in the future?

abn commented 2 years ago

Not sure if it is a limitation, but rather a lack of standards around this. Poetry's source support is based on the PEP 503 standards, which iirc, does not define the distribution of direct link or vcs sources.

Also, since Poetry will need to download (in your case clone) all the package versions separately and inspect the metadata to generate the universal lock file, such a mechanism of consuming packages might be rather inefficient. Additionally, this can also lead to a lot of ambuity. If a package distributed via such a mecahnism is locked, do we identify it as a VCS sourced dependency or a simple API repository sourced dependency? And what happens when a user consumes this lockfile? Does Poetry go over these again and inspect on their local machine? Or do we treat it as VCS dependency and manage it so?

As things are today, I doubt this will be something we will support. But maybe if there are enough folks out there who want something similar, you could potentially create a plugin.

lecardozo commented 2 years ago

Thanks for the clarification!

If a package distributed via such a mecahnism is locked, do we identify it as a VCS sourced dependency or a simple API repository sourced dependency? And what happens when a user consumes this lockfile? Does Poetry go over these again and inspect on their local machine? Or do we treat it as VCS dependency and manage it so?

I tend to think that this should be identified as a VCS sourced dependency instead of Simple API (even though the API is the intermediate here). The API ends up being just a "shortcut" index, so the user doesn't need to explicitly pass the git dependency.

With the link-to-VCS API

[tool.poetry.dependencies]
python = "^3.8"
package = "2.0.0" # still comes from VCS, but has an intermediate API that links to the actual repo

Directly to VCS

[tool.poetry.dependencies]
python = "^3.8"
package = { git = "https://github.com/company/package-repo.git", tag = "2.0.0" } #  points directly to the repo

As things are today, I doubt this will be something we will support. But maybe if there are enough folks out there who want something similar, you could potentially create a plugin.

Yeah, that makes sense! In our case, linking explicitly to the VCS seems to be enough. Although this is a little less convenient from the user's point-of-view, this seems a better choice. 😃

abn commented 2 years ago

@lecardozo there is general interest in supporting something similar to "find-links" in pip. Might be a good discussion topic. Although what you have is more similar to a constraints file.

But all of this seems to be a way to avoid having a proper package index. Having one will also save developers wheel build times too among other things.

lecardozo commented 2 years ago

@lecardozo there is general interest in supporting something similar to "find-links" in pip.

It seems like my use case falls between the find-links and the PEP 503 index approach. I guess a find-links functionality would probably work for this case but doesn't seem ideal.

But all of this seems to be a way to avoid having a proper package index. Having one will also save developers wheel build times too among other things.

Agreed. Although I feel that, in some cases, setting up a proper private package index might be prohibitive/not worth the effort.

abn commented 2 years ago

GitHub packages supporting Python packages might be helpful in your case. Alternatively, I could also recommend that you use a github release action, similar to

https://github.com/python-poetry/poetry/blob/0d418fc00eab4f15ac7e13063369f85f50aba901/.github/workflows/release.yml#L36-L42

And then in your html file link to the release assets, like in https://github.com/python-poetry/poetry/releases/download/1.2.0b1/poetry-1.2.0b1-py3-none-any.whl.

github-actions[bot] commented 6 months ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

python-poetry / poetry