poetry self and custom sources

[colindean]

• [X] I have searched the issues of this repo and believe that this is not a duplicate.
• [X] I have searched the FAQ and general documentation and believe that my question is not already covered.

Feature Request

I'm preparing for the eventuality of the export plugin not being installed by default. When running poetry self add poetry-plugin-export, I noticed that the files were being downloaded from pythonhosted.org instead of my company's repos. I'd prefer that poetry retrieve artifacts from my company's internal repos.

Expectation

⭐ ⭐ ⭐ ⭐ ⭐ = A command line option would be great:

poetry self add poetry-plugin-export --index-url https://example.com/proxies/pypi/simple

⭐ ⭐ ⭐ ⭐ = But I could also see this being a one-time setup thing:

poetry self config pip.global.index-url https://example.com/proxies/pypi/simple

⭐ ⭐ ⭐ = but this probably works for now until something nicer exists:

${POETRY_HOME}/bin/pip3 config --site set global.index-url https://example.com/proxies/pypi/simple

But I don't have a good way for poetry to tell me its venv path for scripting, e.g. something like

POETRY_HOME=$(poetry self venv)

or to run something in poetry's venv with a command like

poetry self run pip config …

Related:

• https://github.com/python-poetry/poetry/pull/4944
• https://github.com/python-poetry/poetry/issues/6529

[neersighted]

Keep in mind that pip is an implementation detail that we are slowly removing from the codebase, so any attempt to support this cannot couple to any details of pip directly.

I think in general the aim is going to be allowing plugins to be installed outside the main Poetry environment (e.g. somehow have a required-plugins section in pyproject.toml) and make them use the sources configured there. But some facilities to manage sources for the global/instance pyproject.toml certainly wouldn't hurt either.

[colindean]

allowing plugins to be installed outside the main Poetry environment (e.g. somehow have a required-plugins section in pyproject.toml

This is definitely the more desirable solution for me. I think we discussed this on Discord a while back!

Maybe this invocation would adequately hide pip:

poetry self config plugins.index-url https://example.com/proxies/pypi/simple

but definitely having a tool.poetry.plugins that accepts dependency specs like tool.poetry.dependencies would be great.

[neersighted]

I don't think your last suggestion is sufficiently abstracted either -- we simply don't have infrastructure for a re-implementation of index-url in Poetry (even now, we actually download packages for pip with our own logic and only use pip to install files from disk), so what you really want is full-resolution source config like any other pyproject.toml.

I actually think that it may even be supported already if you manually edit the file -- but some sort of facility to surface and discover this is still needed.

[colindean]

I tested and can confirm that adding the source to the poetry-instance pyproject.toml correctly shifts poetry to using the proxy.

# $ cat /Users/colin/Library/Application\ Support/pypoetry/pyproject.toml 
[tool.poetry]
name = "poetry-instance"
version = "1.2.1"
description = ""
authors = []
license = ""
[tool.poetry.dependencies]
python = "3.9.11"
poetry = "1.2.1"
[[tool.poetry.source]]
name = "pypi-mirror"
url = "https://stuff.example.com/artifactory/api/pypi/pypi-remote/simple"
default = true
[tool.poetry.group.additional.dependencies]
poetry-plugin-export = "^1.1.1"

The plugins are then retrieved as I'd expect when running poetry self add <plugin>.

Maybe the self command hierarchy could get its own source add|show|remove subcommands.

[colindean]

Took at stab in #6708 at poetry self source add|remove|show and it wffm like this.

I think the long-term solution is to allow specifying plugins inside a project-level pyproject.toml and specifying sources for those plugins just like dependencies, but how I've got this wired up in that PR meets my security compliance needs!

[jonathanunderwood]

I tested and can confirm that adding the source to the poetry-instance pyproject.toml correctly shifts poetry to using the proxy.

When hitting the same challenge, I don't see the reordering you describe actually fix the problem. With the re-ordering I see the initial lookup for the package is done against the local source, but then subsequent dependency resolution seems to be happening against upstream PyPi.

[neersighted]

You appear to be talking about something else completely @jonathanunderwood; I'd suggest reaching out on Discord or Discussions to see if we can puzzle out where the mismatch is.

[jonathanunderwood]

I am really talking about the same problem described by the OP. What makes you think I am talking about something different? (Genuine question - I realize I've caused confusion, but can't quite fathom out how/where).

[neersighted]

There is nothing related to ordering here -- nor have you explained any concrete details of the problem you are having. Plugins work no different from any other pyproject.toml, so this issue is really just an ask for the poetry self source commands and does not track anything unique beyond that, in any case.

[jonathanunderwood]

As I stated: I am seeing precisely and exactly the problem the OP described (hence no need to repeat the description).

In this thread, the OP also reported (in the comment I quoted) that re-ordering the pyproject.toml file to move the [[tool.poetry.source]] section earlier fixed the problem for them. My comment stated that I do not see that course of action fixing the problem for me locally, as later dependency resolution is still going out to PyPi, regardless of the ordering of the pyproject.toml file.

So I am a bit bewildered by the statement "There is nothing related to ordering here".

Anyway, I'll open a different issue with more information when I have time.

[neersighted]

There is still nothing related to order in that comment -- it's referring to adding a [[tool.poetry.source]] record to the internal pyproject.toml (the poetry-instance project). The OP's original confusion was over how to configure the sources used for poetry self commands, and they never tried editing the poetry-instance project until that comment. A source with default = true is not sensitive to order, in any case.

[jonathanunderwood]

OK - gottit, I did indeed misunderstand. Thanks for bearing with me.

Nonetheless, the end result is that I am still seeing an issue (though it might be a different root cause) resulting in the inability to install plugins from a local pypi mirror due to poetry insisting on doing dependency resolution against PyPi for self add.

pyproject.toml:

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

[tool.poetry]
name = "ecba-scoring-api"
version = "0.0.0.post4.dev0+bb465c1"
description = "Scoring API for the ECBA framework"
authors = ["Jonathan G. Underwood <jonathan.underwood@someemail.com>"]
license = "Proprietary"
readme = "README.md"

[[tool.poetry.source]]
name = "pypi-mirror"
url = "https://artifactory.internal.someenterprise.com/artifactory/api/pypi/pypi-python-org/simple"
default = true
secondary = false

[tool.poetry.dependencies]
python = ">=3.10,<3.11"
fastapi = "^0.86.0"
uvicorn = "^0.19.0"
ecba = {git = "https://github.localenterpriseinstance.com/ECBA/pyecba.git", branch = "ecba-scoring-api"}

[tool.poetry.group.dev.dependencies]
flake8 = "^5.0.4"
black = "^22.10.0"
isort = "^5.10.1"
mypy = "^0.990"
pylint = "^2.15.5"
pydocstyle = "^6.1.1"

$ poetry self add "poetry-dynamic-versioning[plugin]"
Using version ^0.21.1 for poetry-dynamic-versioning

Updating dependencies
Resolving dependencies... (12.5s)

HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Max retries exceeded with url: /packages/3d/1a/d31fce69c119df1fddab3706b63c53d363982c55d841d9c3839b12f15327/shellingham-1.5.0-py2.py3-none-any.whl (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:997)')))

So, you can see, poetry self add isn't paying attention to the [[tool.poetry.source]] section, and is trying to reach out to PyPi (and failing to do so due to an SSL error caused by a proxy - that's fine, and expected).

[neersighted]

poetry self add is not respecting your source as you have configured your project to use your proxied source, not your Poetry instance. We are referring to the internal pyproject.toml used by Poetry to manage itself, located at ~/.config/pypoetry/pyproject.toml on a Linux system.

When adding/removing plugins with poetry self, Poetry is used to manage Poetry. You have been confused to believe that project-level source configuration has any level of influence over Poetry's global state; it does not.

If you're still confused by this, please reach out via another channel (e.g. Discord) for assistance as we're just generating noise on this ticket at this point.

[waketzheng]

Before this pull request merged, set mirror for poetry self command can be (poetry version >= 1.5):

python -c "url='http://my_pypi.com';from poetry.locations import CONFIG_DIR;import os;cmd='cd {}&&poetry source remove pypi-mirror&&poetry source add --priority=default pypi-mirror {}'.format(CONFIG_DIR,url);os.system(cmd)"