Open keriksson-rosenqvist opened 1 year ago
Try running poetry update -vvv
and post the output (remember to redact anything you don't want public - like your feed address)
Does pip install -i https://[REDACTED]/pypi/simple/my-private-package
fail with a 401?
@kristang I've made an MWE using a new hello-world
package in my private pipy registry. Running poetry update -vvv
gives the following trace:
Loading configuration file [REDACTED]/.config/pypoetry/config.toml
Loading configuration file [REDACTED]/.config/pypoetry/auth.toml
Adding repository hello_world (https://<username:password>@[REDACTED]/hello_world/pypi/simple) and setting it as secondary
Using virtualenv: [REDACTED]/poetry_credential_mwe/.venv
Project environment contains an empty path in sys_path, ignoring.
Updating dependencies
Resolving dependencies...
1: fact: poetry-credential-mwe is 0.1.0
1: derived: poetry-credential-mwe
1: fact: poetry-credential-mwe depends on hello-world (^0.1.0)
1: selecting poetry-credential-mwe (0.1.0)
1: derived: hello-world (>=0.1.0,<0.2.0)
Creating new session for <username:parrword>@[REDACTED]
[urllib3.connectionpool] Starting new HTTPS connection (1): [REDACTED]:443
[urllib3.connectionpool] https://[REDACTED]:443 "GET [REDACTED]/hello_world/pypi/simple/hello-world/ HTTP/1.1" 200 None
Source (hello_world): 1 packages found for hello-world >=0.1.0,<0.2.0
[urllib3.connectionpool] https://[REDACTED]:443 "GET [REDACTED]/hello_world/pypi/simple/hello-world/ HTTP/1.1" 200 None
Source (hello_world): Downloading wheel: hello_world-0.1.0-py3-none-any.whl
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
[keyring.backend] Loading chainer
[keyring.backend] Loading libsecret
[keyring.backend] Loading macOS
Creating new session for [REDACTED]
[urllib3.connectionpool] Starting new HTTPS connection (1): [REDACTED]:443
[urllib3.connectionpool] https://[REDACTED]:443 "GET [REDACTED]/pypi/download/hello-world/0.1/hello_world-0.1.0-py3-none-any.whl HTTP/1.1" 401 343
1: Version solving took 1.522 seconds.
1: Tried 1 solutions.
ValueError
Package('hello-world', '0.1') is not in list
at ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/poetry/repositories/legacy_repository.py:51 in package
47│ Note that this will be cached so the subsequent operations
48│ should be much faster.
49│ """
50│ try:
→ 51│ index = self._packages.index(Package(name, version))
52│
53│ return self._packages[index]
54│ except ValueError:
55│ package = super().package(name, version, extras)
The following error occurred when trying to handle this error:
Stack trace:
31 ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/cleo/application.py:329 in run
327│
328│ try:
→ 329│ exit_code = self._run(io)
330│ except Exception as e:
331│ if not self._catch_exceptions:
30 ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/poetry/console/application.py:185 in _run
183│ self._load_plugins(io)
184│
→ 185│ exit_code: int = super()._run(io)
186│ return exit_code
187│
29 ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/cleo/application.py:423 in _run
421│ io.input.set_stream(stream)
422│
→ 423│ exit_code = self._run_command(command, io)
424│ self._running_command = None
425│
28 ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/cleo/application.py:465 in _run_command
463│
464│ if error is not None:
→ 465│ raise error
466│
467│ return event.exit_code
27 ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/cleo/application.py:449 in _run_command
447│
448│ if event.command_should_run():
→ 449│ exit_code = command.run(io)
450│ else:
451│ exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED
26 ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/cleo/commands/base_command.py:119 in run
117│ io.input.validate()
118│
→ 119│ status_code = self.execute(io)
120│
121│ if status_code is None:
25 ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/cleo/commands/command.py:83 in execute
81│
82│ try:
→ 83│ return self.handle()
84│ except KeyboardInterrupt:
85│ return 1
24 ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/poetry/console/commands/update.py:54 in handle
52│ self.installer.update(True)
53│
→ 54│ return self.installer.run()
55│
23 ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/poetry/installation/installer.py:114 in run
112│ self._execute_operations = False
113│
→ 114│ return self._do_install()
115│
116│ def dry_run(self, dry_run: bool = True) -> Installer:
22 ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/poetry/installation/installer.py:247 in _do_install
245│ source_root=self._env.path.joinpath("src")
246│ ):
→ 247│ ops = solver.solve(use_latest=self._whitelist).calculate_operations()
248│ else:
249│ self._io.write_line("Installing dependencies from lock file")
21 ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/poetry/puzzle/solver.py:73 in solve
71│ with self._provider.progress():
72│ start = time.time()
→ 73│ packages, depths = self._solve(use_latest=use_latest)
74│ end = time.time()
75│
20 ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/poetry/puzzle/solver.py:151 in _solve
149│
150│ try:
→ 151│ result = resolve_version(
152│ self._package, self._provider, locked=locked, use_latest=use_latest
153│ )
19 ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/poetry/mixology/__init__.py:24 in resolve_version
22│ solver = VersionSolver(root, provider, locked=locked, use_latest=use_latest)
23│
→ 24│ return solver.solve()
25│
18 ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/poetry/mixology/version_solver.py:127 in solve
125│ while next is not None:
126│ self._propagate(next)
→ 127│ next = self._choose_package_version()
128│
129│ return self._result()
17 ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/poetry/mixology/version_solver.py:446 in _choose_package_version
444│ package = locked
445│
→ 446│ package = self._provider.complete_package(package)
447│
448│ conflict = False
16 ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/poetry/puzzle/provider.py:530 in complete_package
528│ dependency_package = DependencyPackage(
529│ dependency,
→ 530│ self._pool.package(
531│ package.pretty_name,
532│ package.version,
15 ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/poetry/repositories/pool.py:152 in package
150│
151│ if repository is not None and not self._ignore_repository_names:
→ 152│ return self.repository(repository).package(name, version, extras=extras)
153│
154│ for repo in self._repositories:
14 ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/poetry/repositories/legacy_repository.py:55 in package
53│ return self._packages[index]
54│ except ValueError:
→ 55│ package = super().package(name, version, extras)
56│ package._source_type = "legacy"
57│ package._source_url = self._url
13 ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/poetry/repositories/cached.py:86 in package
84│ extras: list[str] | None = None,
85│ ) -> Package:
→ 86│ return self.get_release_info(canonicalize_name(name), version).to_package(
87│ name=name, extras=extras
88│ )
12 ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/poetry/repositories/cached.py:63 in get_release_info
61│ return PackageInfo.load(self._get_release_info(name, version))
62│
→ 63│ cached = self._cache.remember_forever(
64│ f"{name}:{version}", lambda: self._get_release_info(name, version)
65│ )
11 ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/cachy/repository.py:174 in remember_forever
172│ return val
173│
→ 174│ val = value(callback)
175│
176│ self.forever(key, val)
10 ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/cachy/helpers.py:6 in value
4│ def value(val):
5│ if callable(val):
→ 6│ return val()
7│
8│ return val
9 ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/poetry/repositories/cached.py:64 in <lambda>
62│
63│ cached = self._cache.remember_forever(
→ 64│ f"{name}:{version}", lambda: self._get_release_info(name, version)
65│ )
66│
8 ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/poetry/repositories/legacy_repository.py:121 in _get_release_info
119│ yanked = page.yanked(name, version)
120│
→ 121│ return self._links_to_data(
122│ links,
123│ PackageInfo(
7 ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/poetry/repositories/http.py:256 in _links_to_data
254│ data.files = files
255│
→ 256│ info = self._get_info_from_urls(urls)
257│
258│ data.summary = info.summary
6 ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/poetry/repositories/http.py:193 in _get_info_from_urls
191│ # Prefer non platform specific wheels
192│ if universal_python3_wheel:
→ 193│ return self._get_info_from_wheel(universal_python3_wheel)
194│
195│ if universal_python2_wheel:
5 ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/poetry/repositories/http.py:84 in _get_info_from_wheel
82│ with temporary_directory() as temp_dir:
83│ filepath = Path(temp_dir) / filename
→ 84│ self._download(url, filepath)
85│
86│ return PackageInfo.from_wheel(filepath)
4 ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/poetry/repositories/http.py:72 in _download
70│
71│ def _download(self, url: str, dest: Path) -> None:
→ 72│ return download_file(url, dest, session=self.session)
73│
74│ def _get_info_from_wheel(self, url: str) -> PackageInfo:
3 ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/poetry/utils/helpers.py:85 in download_file
83│ get = requests.get if not session else session.get
84│
→ 85│ response = get(url, stream=True, timeout=REQUESTS_TIMEOUT)
86│ response.raise_for_status()
87│
2 ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/poetry/utils/authenticator.py:246 in get
244│
245│ def get(self, url: str, **kwargs: Any) -> requests.Response:
→ 246│ return self.request("get", url, **kwargs)
247│
248│ def post(self, url: str, **kwargs: Any) -> requests.Response:
1 ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/poetry/utils/authenticator.py:232 in request
230│ if resp.status_code not in [502, 503, 504] or is_last_attempt:
231│ if raise_for_status:
→ 232│ resp.raise_for_status()
233│ return resp
234│
HTTPError
401 Client Error: Unauthorized for url: https://[REDACTED]/pypi/download/hello-world/0.1/hello_world-0.1.0-py3-none-any.whl#sha256=[REDACTED]
at ~/.local/share/pypoetry/venv/lib/python3.9/site-packages/requests/models.py:1021 in raise_for_status
1017│ f"{self.status_code} Server Error: {reason} for url: {self.url}"
1018│ )
1019│
1020│ if http_error_msg:
→ 1021│ raise HTTPError(http_error_msg, response=self)
1022│
1023│ def close(self):
1024│ """Releases the connection back to the pool. Once this method has been
1025│ called the underlying ``raw`` object must not be accessed again.
Running poetry run pip install -i https://[REDACTED]/hello_world/pypi/simple hello-world
without providing any user credentials also fails with a 401:
$ poetry run pip install -i https://[REDACTED]/hello_world/pypi/simple hello-world
Looking in indexes: https://[REDACTED]/hello_world/pypi/simple
User for [REDACTED]:
WARNING: 401 Error, Credentials not correct for https://[REDACTED]/hello_world/pypi/simple/hello-world/
ERROR: Could not find a version that satisfies the requirement hello-world (from versions: none)
ERROR: No matching distribution found for hello-world
User for [REDACTED]:
WARNING: 401 Error, Credentials not correct for https://[REDACTED]/hello_world/pypi/simple/pip/
Only thing I find a little odd is the url returned: Unauthorized for url: https://[REDACTED]/pypi/download/hello-world/0.1/hello_world-0.1.0-py3-none-any.whl#sha256=[REDACTED]
I don't recall seeing that "/download/" part when I install from a private source. What service is providing your private feed?
It might just be my inexperience with pypi feeds in general.
I am using Azure Artifacts to store my private feed. As mentioned it works fine with pip install -i
on the command line when providing credentials. If I supply the -vvv
flag to my pip command it does use that same url https://[REDACTED]/pypi/download/hello-world/0.1/hello_world-0.1.0-py3-none-any.whl#sha256=[REDACTED]
Have you tried other authentication methods? I believe Azure Artifacts supports some kind of API token - last time I checked they had a guide somewhere included on the site for your feed on how to use it.
@kristang That is what's being used. The tokens are used through the basic auth approach. They have no support for other authentications through pip as far as I have found, and I've been in touch with their support team regarding authentication methods.
According to the documentation from Microsoft you can auth with artifacts-keyring
.
You can then install with:
pip install <package-name> --index-url https://pkgs.dev.azure.com/<your-organization-name>/<your-project-name>/_packaging/<your-feed-name>/pypi/simple
This approach worked for me when I was testing Azure Artifacts. There is also support for using a Personal Access Token, which I assume is what you have been trying to do?
A keyring would not solve the problem I've defined:
I am trying to set up a poetry environment which pulls a certain package from a private pypi-like registry. As I am sharing the project with others via a git repository, I want the setup to be contained within the pyproject.toml file, i.e. not rely on configs, environment variables, or keyring settings that may differ between user devices.
With regards to the API token versus PAT, the documentation shows that to use the pip
interface you must use a PAT. I have not found any documentation on another form of token that could be used for this.
Arh, sorry I completely skipped that.
(Off-topic and sidenote: I would be careful about storing your credentials like that in your git)
I'm facing similar problem which happened on a self-host GitLab PyPI repo too.
On version 1.2.2
, poetry lock
failed, but it works on version 1.1.15
.
I'm currently running version 1.2.1, but if this used to work in an older version I'd say it's definitely a bug that it's not working now.
same problem here..
@squirrel532 with version 1.1.15 poetry lock
works, but installation still fails for me.
It's interesting that the authentication seems to work in the first step. Depending on which version I set in the pyproject.toml (e.g. ^0.4.0, <=0.4.0) it is able to figure out which package it should install including the correct download link. But then in the second step when it comes to downloading the package, the authentication fails.
Is there any further update on this? Can it be tagged as a bug please?
As a workaround, I configured pip
with .netrc
, which contains the credentials of my private repo.
https://pip.pypa.io/en/stable/topics/authentication/#netrc-support
As a workaround, I configured
pip
with.netrc
, which contains the credentials of my private repo.https://pip.pypa.io/en/stable/topics/authentication/#netrc-support
How do you set that up within the git repo/pyproject file? The issue I am trying to resolve revolves around the ability to have the poetry environment contain all the information needed to install the whole setup. Any settings or credentials needed outside the git/poetry files will not solve the problem.
I spent the day trying to figure out how to circumvent this issue. In the end, the only thing that works is using a git
source:
Instead of:
[tool.poetry.dependencies]
mydep = { source = "gitlab", version = "0.1.0" }
[[tool.poetry.source]]
name = "gitlab"
url = "https://username:password@gitlab.com/api/v4/projects/1234/packages/pypi/simple/"
secondary = true
I used:
[tool.poetry.dependencies]
mydep = { git = "https://username:password@gitlab.com/myorg/mydep.git", tag = "0.1.0" }
This completely bypass the pypi server but there seems to be no other choice while this issue is not fixed.
Another workaround that works for me (using a gitlab private package repository), is to put the credentials in the project poetry.toml
:
[http-basic.gitlab-mygroup]
username = "gitlab-deploy-token-XX"
password = "<password>"
With gitlab-mygroup
being a source defined in the project pyproject.toml
.
@keriksson-rosenqvist
You should store your credentials in poetry.toml
like @juleswh and remove credentials from pyproject.toml
. Here is my config that works on version 1.3.2
.
# pyproject.toml
[[tool.poetry.source]]
name = "gitlab"
url = "https://self-host-gitlab.example.com/pypi/simple/"
secondary = true
# poetry.toml
[http-basic.gitlab]
username = "gitlab_user"
password = "access_token"
I traced the code and found that poetry
uses .netloc of the .whl
file URL to obtain credentials that stores in [http-basic.xxx]
. It fails to find the credentials since "username:password@gitlab.example.com" != "gitlab.example.com"
.
For someone cares about the code, here is the pseudo calling stack:
poetry/installation/executor.py:707 _download
poetry/installation/executor.py:717 _download_link
poetry/installation/executor.py:758 _download_archive
poetry/utils/authenticator.py:219 request
poetry/utils/authenticator.py:342 get_credentials_for_url
poetry/utils/authenticator.py:300 _get_credentials_for_url
poetry/utils/authenticator.py:421 _get_repository_config_for_url
if repository.netloc == parsed_url.netloc:
parsed_url
is what GitLab Pypi simple API returned, which does not contain credentials.
@squirrel532
Thank you for the suggested work around. Unfortunately I am unable to recreate this working successfully. I can use my generated token to install my package with poetry run pip install -i https://<username>:<token>@<url>
but when following your suggested setup with pyproject.toml and poetry.lock I keep getting Authorization error accessing https://<url>
.
Are there any special cases in the setup that could be causing this? E.g. my source name includes an underscore. Should the addition to the poetry file be done at a specific location?
# pyproject.toml
[tool.poetry.dependencies]
python = "^3.9"
hello-world = { version = "^0.1.0", source = "hello_world"}
[[tool.poetry.source]]
name = "hello_world"
url = "https://[REDACTED]/pypi/simple/"
secondary = true
# poetry.lock
[http-basic.hello_world]
username = "<username>"
password = "<token>"
Execution output
$ poetry update -vvv
Loading configuration file [REDACTED]/pypoetry/config.toml
Loading configuration file [REDACTED]/pypoetry/auth.toml
Adding repository hello_world (https://[REDACTED]/pypi/simple) and setting it as secondary
Using virtualenv: [REDACTED]/.venv
Updating dependencies
Resolving dependencies...
1: fact: poetry-credential-mwe is 0.1.0
1: derived: poetry-credential-mwe
1: fact: poetry-credential-mwe depends on hello_world (^0.1.0)
1: selecting poetry-credential-mwe (0.1.0)
1: derived: hello-world (>=0.1.0,<0.2.0)
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
[keyring.backend] Loading chainer
[keyring.backend] Loading libsecret
[keyring.backend] Loading macOS
Creating new session for [REDACTED]
[urllib3.connectionpool] Starting new HTTPS connection (1): [REDACTED]:443
[urllib3.connectionpool] https://[REDACTED]:443 "GET [REDACTED]/simple/hello-world/ HTTP/1.1" 401 343
Source (hello_world): Authorization error accessing https://[REDACTED]/pypi/simple/hello-world/
Source (hello_world): No packages found for hello-world
Source (hello_world): 0 packages found for hello-world >=0.1.0,<0.2.0
Falling back to installed packages to discover metadata for hello-world
Found 0 compatible packages for hello-world
1: fact: no versions of hello-world match >=0.1.0,<0.2.0
1: conflict: no versions of hello-world match >=0.1.0,<0.2.0
1: ! hello-world (>=0.1.0,<0.2.0) is satisfied by hello-world (>=0.1.0,<0.2.0)
1: ! which is caused by "poetry-credential-mwe depends on hello-world (^0.1.0)"
1: ! thus: version solving failed
1: Version solving took 0.400 seconds.
1: Tried 1 solutions.
....
@kristang The second file should be poetry.toml
, not poetry.lock
.
@squirrel532 Thank you so much. I was staring myself blind there. This works great!
This workaround works but I don’t think the issue should be closed because the original problem still exists.
@bfontaine I think it is risky that someone places their credentials in pyproject.toml
. Maybe we just places a warning in document to prohibit this behavior ?
CC @sdispater
I don’t see any technical reason this shouldn’t work. Right now this is a bug: the URL including credentials is perfectly valid but Poetry appears to silently strip the credentials.
It’s risky for public packages, but there are valid use-cases in a private/company environment. See what @keriksson-rosenqvist wrote in their initial message:
I want the setup to be contained within the pyproject.toml file, i.e. not rely on configs, environment variables, or keyring settings that may differ between user devices.
Hi! I also experience this issue. Thanks to @squirrel532 for this work around. So we can keep our internal projects going.
I think, people should be careful with their credentials and think twice before putting them anywhere. But in our internal projects at work it is quite useful to checkout private repos which are dependent on each other.
So if I understood it correctly the bug is still there but will not be fixed as the issue was closed? 🤔
Since there seems to be more people experiencing this issue, I've reopened it in hopes the bug will be looked at in the future.
I cannot imagine the project will ever accept credentials in pyproject.toml, poetry has no idea where that pyproject.toml will be published, in public or private. The first project that accidentally publishes credentials in pyproject.toml will end up with a security bug against poetry.
@clintonroy The question is not if the project accepts credentials because it doesn’t have a choice: URLs that contain credentials are valid HTTP URLs that shouldn’t be dealt with in a different manner than others. The suggested solution is to put the credentials in poetry.toml
, which you can put in git as well. I don’t remember seeing a security bug against SSH because someone put their private key in a public Git repository.
Hi!, thanks a lot for the instructions above!
Just wanted to mention, that we can add credentials without creating additional poetry.toml
locally.
Here is the config command:
poetry config http-basic.your_repo_alias "username" "your_token"
So such steps need:
Add credentials as mentioned
pyproject.toml
edit like squirrel532 provided above:
[tool.poetry.dependencies]
you_awesome_package = { source = "your_repo_alias", version = "^X.X.X" }
[[tool.poetry.source]]
name = "your_repo_alias"
url = "https://cloud-or-own-gitlab.tld/api/v4/projects/<PROJECT_ID_HERE>/packages/pypi/simple"
secondary = true
Use poetry add you_awesome_package
or poetry update you_awesome_package
as usual.
@fannigurt yes, but this issue is specifically about adding credentials inside the repository in order to share them with everyone. By default poetry config
edits your user config. To edit the local config, you can use poetry config --local
and get the same result as the workaround mentioned earlier.
This issue is still present in Poetry 1.5.1
, and I believe that I've gotten to the bottom of it.
In my testing with Google Artifact Registry (GAR), I'm seeing that the simple repository base URL is returning href
values with absolute URLs. In my testing, if relative URLs are returned, there is no issue. I believe that it is likely that everyone experiencing this issue is using a repository which returns absolute URLs.
<!DOCTYPE html>
<html>
<head>
<title> Links for [REDACTED] </title>
</head>
<body>
<h1> Links for [REDACTED] </h1>
<a href="https://us-east1-python.pkg.dev/[REDACTED]-py3-none-any.whl#sha256=deadbeef">[REDACTED]-py3-none-any.whl</a><br/>
<a href="https://us-east1-python.pkg.dev/[REDACTED].tar.gz#sha256=feedface">[REDACTED].tar.gz</a><br/>
</body>
</html>
PEP 503 states:
URLs may be either absolute or relative as long as they point to the correct location.
So there's nothing invalid about GAR's response.
However, within poetry, this does not play nicely with the HTTP basic authentication credentials. When poetry processes these responses, urllib.parse.urljoin
is used to combine the source's authenticated URL with the non-authenticated URLs from the HTML above.
https://github.com/python-poetry/poetry/blob/dfd0db8366301087bcb1b596e0344cd0450448e5/src/poetry/repositories/link_sources/html.py#L35
urljoin
drops the credentials when joining the two URLs.
import urllib.parse
url='https://username:password@us-east1-python.pkg.dev/path/to/repo/'
href='https://us-east1-python.pkg.dev/path/to/repo/[REDACTED].tar.gz#sha256=feedface'
print(urllib.parse.urljoin(url, href))
Produces: https://us-east1-python.pkg.dev/path/to/repo/[REDACTED].tar.gz#sha256=feedface
7486 is perhaps the best solution here, as it would intercept the credentials into Poetry's authentication system early on before they have a chance to get mangled.
However, there are some other potential risks caused by the same issue. For example, a user could be connecting to a repository via a proxy, and the returned absolute path would override the proxy URL.
import urllib.parse
url='https://username:password@my.proxy.com/path/to/repo/'
href='https://us-east1-python.pkg.dev/path/to/repo/[REDACTED].tar.gz#sha256=feedface'
print(urllib.parse.urljoin(url, href))
Produces: https://us-east1-python.pkg.dev/path/to/repo/[REDACTED].tar.gz#sha256=feedface
TLDR: check your ~/.netrc
file for conflicting credentials.
I think I am having the same/similar issue. Unfortunately the workarounds did not work in my case. I believe in my case the error has to do with my environment (i.e. the installed packages, libraries, etc.). I'm on Ubuntu 22.04 using Poetry 1.6.1. AFAIK I haven't made any changes to poetry credential configuration between the time everything was working (~1 week ago) and now.
I have traced my error to my system because (after a lot of troubleshooting), I found that running poetry commands (lock
, update
, install
) in a docker container and using a secret mount to bind a local auth.toml
to ~/.config/pypoetry/auth.toml
in the container works to authenticate and pull the packages from the private repo. Also, putting the credentials directly into the source URL works but due to the behavior described in https://github.com/python-poetry/poetry/issues/6799#issuecomment-1418496011 (as well as the obvious security risks with storing secrets in distributable code) this is not a full workaround.
Therefore I know that a) the credentials are valid and b) on a "clean" system the process works. At some point some package or library poetry uses to authenticate must have gotten corrupted in my local system however attempts to hunt this package down have all failed - downgrading poetry, force installing python and system packages like keyring, urllib, python-dbus, libdbus, etc. have all failed to solve the problem.
Additionally disabling the keyring entirely and either falling back to plaintext or environment variable authentication does not work either. Unfortunately, due to the nature of the problem I cannot produce a reproduction script since I have no idea which library on my system has become "corrupted". I'm hoping someone can at least point me in a direction to look.
Any help would be greatly appreciated. In the meantime I am about to run the poetry lock
command through a debugger to introspect what exact requests and metadata are being sent/received so hopefully I can isolate whether the issue lies in poetrys ability to retrieve the credentials from the keyring/plaintext/environment or a malformed request itself.
I've been able to rule out poetry not getting the credentials from the keyring - it gets the credentials, encrypts them, and places them into the header. I haven't checked if the encryption is correct yet. The content of the response claims the token has expired but this doesn't make sense as the repository maintainers checked this and confirmed it was valid. Furthermore the issue occurs with my personal username and token too (I have developer access to the repository).
Just reinstalled requests and cryptography too.
Okay - I finally traced the error to conflicting credentials in ~/.netrc
vs the keyring. Apparently, poetry
will use requests
to get credentials for the repository (see here) which uses ~/.netrc
credentials to form the encoded authorization in the header of the request. So any changes to the configuration of the credentials using poetry config
, environment variables, etc. is essentially discarded for authenticating the actual request in favor of the contents of ~/.netrc
SO if you come across this issue and are in a situation like mine - where none of the above workarounds fixed anything - check your .netrc
file for configurations matching the repository URL and either remove them or modify them so poetry uses the correct credentials.
I think I am having the same/similar issue. Unfortunately the workarounds did not work in my case. I believe in my case the error has to do with my environment (i.e. the installed packages, libraries, etc.). I'm on Ubuntu 22.04 using Poetry 1.6.1. AFAIK I haven't made any changes to poetry credential configuration between the time everything was working (~1 week ago) and now.
I have traced my error to my system because (after a lot of troubleshooting), I found that running poetry commands (
lock
,update
,install
) in a docker container and using a secret mount to bind a localauth.toml
to~/.config/pypoetry/auth.toml
in the container works to authenticate and pull the packages from the private repo. Also, putting the credentials directly into the source URL works but due to the behavior described in #6799 (comment) (as well as the obvious security risks with storing secrets in distributable code) this is not a full workaround.Therefore I know that a) the credentials are valid and b) on a "clean" system the process works. At some point some package or library poetry uses to authenticate must have gotten corrupted in my local system however attempts to hunt this package down have all failed - downgrading poetry, force installing python and system packages like keyring, urllib, python-dbus, libdbus, etc. have all failed to solve the problem.
Additionally disabling the keyring entirely and either falling back to plaintext or environment variable authentication does not work either. Unfortunately, due to the nature of the problem I cannot produce a reproduction script since I have no idea which library on my system has become "corrupted". I'm hoping someone can at least point me in a direction to look.
Any help would be greatly appreciated. In the meantime I am about to run the
poetry lock
command through a debugger to introspect what exact requests and metadata are being sent/received so hopefully I can isolate whether the issue lies in poetrys ability to retrieve the credentials from the keyring/plaintext/environment or a malformed request itself.UPDATE 2024-10-24
I've been able to rule out poetry not getting the credentials from the keyring - it gets the credentials, encrypts them, and places them into the header. I haven't checked if the encryption is correct yet. The content of the response claims the token has expired but this doesn't make sense as the repository maintainers checked this and confirmed it was valid. Furthermore the issue occurs with my personal username and token too (I have developer access to the repository).
Just reinstalled requests and cryptography too.
Okay - I finally traced the error to conflicting credentials in
~/.netrc
vs the keyring. Apparently,poetry
will userequests
to get credentials for the repository (see here) which uses~/.netrc
credentials to form the encoded authorization in the header of the request. So any changes to the configuration of the credentials usingpoetry config
, environment variables, etc. is essentially discarded for authenticating the actual request in favor of the contents of~/.netrc
SO if you come across this issue and are in a situation like mine - where none of the above workarounds fixed anything - check your
.netrc
file for configurations matching the repository URL and either remove them or modify them so poetry uses the correct credentials.
It is 2023-10-24, not 2024
I spent the day trying to figure out how to circumvent this issue. In the end, the only thing that works is using a
git
source:Instead of:
[tool.poetry.dependencies] mydep = { source = "gitlab", version = "0.1.0" } [[tool.poetry.source]] name = "gitlab" url = "https://username:password@gitlab.com/api/v4/projects/1234/packages/pypi/simple/" secondary = true
I used:
[tool.poetry.dependencies] mydep = { git = "https://username:password@gitlab.com/myorg/mydep.git", tag = "0.1.0" }
This completely bypass the pypi server but there seems to be no other choice while this issue is not fixed.
woow, if you create default username for token, which is gitlab+deploy-token-{n}
, poetry will consider it as a wrong git url. I advice to create simple usernames like user
. Thanks for an answer, that helped!
I was having this same issue. Turns out I had my Gitlab credentials stored in the .netrc file as well, so @gresavage solution solved the failed authorization issues for me.
I had a similar issue with GitLab. Since the package I wanted to install was in another repo in GitLab, I had to go to that repo's Settings > CI/CD and add my other repo (the one where the pipelines run) under Token Access > Limit access to this project > Groups and projects with access. More info.
After that, I could use Poetry to add my private package with a pipeline job token.
Discussed in https://github.com/orgs/python-poetry/discussions/6629