Open ckchow opened 8 months ago
dimbleby
commented
8 months ago Perhaps your credentials have permissions in one of those places but not the other
Perhaps one of them is being answered from cache
Unless you can provide a way to reproduce this, it is unlikely that we can guess what you are seeing
ckchow
commented
8 months ago The credentials work on all the URLs listed, I tried them in my browser and was able to download the artifacts.
Probably not answered from cache, this behavior happens in a completely fresh python:3.8 docker container.
I was chatting in discord with abn and they mentioned that it might be possible to repro and test this with a devpi test fixture.
dimbleby
commented
8 months ago Sure, if you can provide a repro that way, that works.
Lots of us use credentials for private repositories all the time, so I have to guess this is likely to be something specific to you.
I think there is a good chance you will find it difficult to set up a repro - but if so, hopefully in trying you will learn something about what it is that is unusual in your environment.
ckchow
commented
8 months ago Maybe it's washed out in the report details, but the fact that PYTHON_KEYRING_BACKEND=keyring.backends.null.Keyring poetry install works does feel like it's a keyring issue. I will see if I can repro in a hermetic environment if I get some free time.
Garett601
commented
8 months ago I can confirm we are also running into private package authentication issues in our CI pipeline runs. We have narrowed it down to poetry 1.8.x being the cause.
When we pin the poetry version to 1.7.1, it installs perfectly fine. But as soon as the version isn't pinned and the pipeline install poetry 1.8.x, then the authorization fails.
Please see below: This fails authentication when poetry 1.8.x is installed during the pipeline run
- task: PipAuthenticate@1
displayName: 'Pip Authentication'
inputs:
artifactFeeds: '<feed_name>'
onlyAddExtraIndex: true
- script: |
curl -sSL https://install.python-poetry.org | python3 -
export PATH=$PATH:$HOME/.poetry/bin
export PAT=$(echo "$PIP_EXTRA_INDEX_URL" | sed -n 's/.*build:\([^@]*\)@.*/\1/p')
poetry config http-basic.<source_name> $PAT ""
poetry install
displayName: 'Install dependencies'
Work around: Pin the version of poetry to 1.7.1
- task: PipAuthenticate@1
displayName: 'Pip Authentication'
inputs:
artifactFeeds: '<feed_name>'
onlyAddExtraIndex: true
- script: |
curl -sSL https://install.python-poetry.org | python3 - --version 1.7.1
export PATH=$PATH:$HOME/.poetry/bin
export PAT=$(echo "$PIP_EXTRA_INDEX_URL" | sed -n 's/.*build:\([^@]*\)@.*/\1/p')
poetry config http-basic.<source_name> $PAT ""
poetry install
displayName: 'Install dependencies'
abn
commented
8 months ago This fails authentication when poetry 1.8.x is installed during the pipeline run
@Garett601 can you please provide the -vvv log please? The "Authorization error" seems to indicate the problem actually lies elsewhere and is a separate issue to that of what is reported here.
PSC-MM
commented
8 months ago We're seeing similar on v1.8.2, when installing locally on a Windows machine, trying to authenticate to an AWS CodeArtifact repo. I think it may have something to do with the length of the token provided by CodeArtifact (but this wasn't an issue with earlier Poetry versions).
If I specify the source as follows in pyproject.toml, then run 'poetry lock', then 'poetry install', I get the 401 issue:
[[tool.poetry.source]] name = "aws" url = "https://ourpypi-123456789012.aws_account_id.d.codeartifact.eu-west-2.amazonaws.com/pypi/ourpypi/simple/" priority = "primary"
I have tried this by passing creds via 'poetry config http-basic.aws aws $token', including with the keychain disabled, and as environment variables.
If instead I set the source as follows in pyproject.toml, then run 'poetry lock', then 'poetry install', it works:
[[tool.poetry.source]] name = "aws" url = "https://aws:[long_CodeArtifact_token]@ourpypi-123456789012.d.codeartifact.eu-west-2.amazonaws.com/pypi/ourpypi/simple/" priority = "primary"
NB The sha256 values of each package in the lock file do not change between the two variants of pyproject.toml following 'poetry lock'. They are valid. Only package.source.url varies.
The verbose logging for the error is:
PS C:\Code\xxx> poetry install -vvv Loading configuration file C:\Users\xxx\AppData\Roaming\pypoetry\config.toml Loading configuration file C:\Users\xxx\AppData\Roaming\pypoetry\auth.toml Adding repository aws (https://ourpypi-123456789012.d.codeartifact.eu-west-2.amazonaws.com/pypi/ourpypi/simple) and setting it as primary Deactivating the PyPI repository Using virtualenv: C:\Code\xxx.venv Installing dependencies from lock file
Finding the necessary packages for the current system
Package operations: 54 installs, 0 updates, 0 removals, 1 skipped
Installing six (1.16.0): Failed
Stack trace:
11 ~\AppData\Roaming\pypoetry\venv\Lib\site-packages\poetry\installation\executor.py:281 in _execute_operation 279│ 280│ try: → 281│ result = self._do_execute_operation(operation) 282│ except EnvCommandError as e: 283│ if e.e.returncode == -2:
10 ~\AppData\Roaming\pypoetry\venv\Lib\site-packages\poetry\installation\executor.py:391 in _do_execute_operation 389│ return 0 390│ → 391│ result: int = getattr(self, f"execute{method}")(operation) 392│ 393│ if result != 0:
9 ~\AppData\Roaming\pypoetry\venv\Lib\site-packages\poetry\installation\executor.py:516 in _execute_install 514│ 515│ def _execute_install(self, operation: Install | Update) -> int: → 516│ status_code = self._install(operation) 517│ 518│ self._save_url_reference(operation)
8 ~\AppData\Roaming\pypoetry\venv\Lib\site-packages\poetry\installation\executor.py:554 in _install 552│ archive = self._download_link(operation, Link(package.source_url)) 553│ else: → 554│ archive = self._download(operation) 555│ 556│ operation_message = self.get_operation_message(operation)
7 ~\AppData\Roaming\pypoetry\venv\Lib\site-packages\poetry\installation\executor.py:746 in _download 744│ self._yanked_warnings.append(message) 745│ → 746│ return self._download_link(operation, link) 747│ 748│ def _download_link(self, operation: Install | Update, link: Link) -> Path:
6 ~\AppData\Roaming\pypoetry\venv\Lib\site-packages\poetry\installation\executor.py:753 in _download_link 751│ # Get original package for the link provided 752│ download_func = functools.partial(self._download_archive, operation) → 753│ original_archive = self._artifact_cache.get_cached_archive_for_link( 754│ link, strict=True, download_func=download_func 755│ )
5 ~\AppData\Roaming\pypoetry\venv\Lib\site-packages\poetry\utils\cache.py:272 in get_cached_archive_for_link 270│ cache_dir.mkdir(parents=True, exist_ok=True) 271│ try: → 272│ download_func(link.url, cached_archive) 273│ except BaseException: 274│ cached_archive.unlink(missing_ok=True)
4 ~\AppData\Roaming\pypoetry\venv\Lib\site-packages\poetry\installation\executor.py:821 in _download_archive 819│ dest: Path, 820│ ) -> None: → 821│ downloader = Downloader(url, dest, self._authenticator) 822│ wheel_size = downloader.total_size 823│
3 ~\AppData\Roaming\pypoetry\venv\Lib\site-packages\poetry\utils\helpers.py:177 in init 175│ headers = {"Accept-Encoding": "Identity"} 176│ → 177│ self._response = get( 178│ url, stream=True, headers=headers, timeout=REQUESTS_TIMEOUT 179│ )
2 ~\AppData\Roaming\pypoetry\venv\Lib\site-packages\poetry\utils\authenticator.py:267 in get 265│ 266│ def get(self, url: str, kwargs: Any) -> requests.Response: → 267│ return self.request("get", url, kwargs) 268│ 269│ def head(self, url: str, **kwargs: Any) -> requests.Response:
1 ~\AppData\Roaming\pypoetry\venv\Lib\site-packages\poetry\utils\authenticator.py:245 in request 243│ if resp.status_code not in STATUS_FORCELIST or is_last_attempt: 244│ if raise_for_status: → 245│ resp.raise_for_status() 246│ return resp 247│
HTTPError
401 Client Error: Unauthorized for url: https://ourpypi-123456789012.d.codeartifact.eu-west-2.amazonaws.com/pypi/ourpypi/simple/six/1.16.0/six-1.16.0-py2.py3-none-any.whl#sha256=8abb2f1d86890a2dfb989f9a77cfcfd3e47c2a354b01111771326f8aa26e0254
at ~\AppData\Roaming\pypoetry\venv\Lib\site-packages\requests\models.py:1021 in raise_for_status
1017│ f"{self.status_code} Server Error: {reason} for url: {self.url}"
1018│ )
1019│
1020│ if http_error_msg:
→ 1021│ raise HTTPError(http_error_msg, response=self)
1022│
1023│ def close(self):
1024│ """Releases the connection back to the pool. Once this method has been
1025│ called the underlying raw object must not be accessed again.
Cannot install six.
Garett601
commented
8 months ago We're seeing similar on v1.8.2, when installing locally on a Windows machine, trying to authenticate to an AWS CodeArtifact repo. I think it may have something to do with the length of the token provided by CodeArtifact (but this wasn't an issue with earlier Poetry versions).
I also thought this was the issue, and checked the PAT length as a debug step, and it did seem like an odd length (But this was the same as before and it works perfectly with poetry 1.7.1)
PSC-MM
commented
8 months ago Just to say that I got past this issue, for the time being, by uninstalling and reinstalling Poetry (current version, 1.8.2) on my Windows machine. I have successfully with both PYTHON_KEYRING_BACKEND=keyring.backends.null.Keyring (as per OP) and poetry config keyring.enabled false. Both result in the creds being stored in plaintext in auth.toml, which is fine for now.
It is just a suspicion and not something I can follow up on, but the trigger for this behaviour may have been when we tried authenticating to CodeArtifact with a token longer than allowed by keychain on Windows, which from one report is 1280 characters. Seems that CodeArtifact tokens are around 1400 chrs when authenticating via a role, vs ~1200 chrs when authenticating via a user. I saw the same behaviour then as described in this issue: https://github.com/python-poetry/poetry/issues/6597. Since then, and until reinstalling, this current issue has been in play.
bcvanmeurs
commented
4 months ago I encounter the same issue in a python:3.11.9 docker container with poetry installed via the install script. It works with poetry 1.7.1 but not with poetry 1.8.3. Any of the above workarounds do not work for me (setting PYTHON_KEYRING_BACKEND and poetry config keyring.enabled false).
peterschutt
commented
3 months ago @Garett601 - perhaps your issue is the same as one that I've encountered. I see in your config you use an empty password:
poetry config http-basic.<source_name> $PAT ""
Gemfury recommends the same pattern (https://gemfury.com/help/errors/repo-url-password/) and my builds also started failing after upgrading to 1.8.
I set the token to the auth user via environment in our dockerfiles, and do not set a password:
# Export fury auth to poetry
ARG fury_auth
ENV POETRY_HTTP_BASIC_FURY_USERNAME ${fury_auth}Seems to originate in e4cf21831d467d1369f52941f56c248ee0466871 where I think it is this change that is responsible for the difference.
Prior, if a username is set but no password, get_http_auth() would always return {"username": "whateverthetokenis", "password": None}, irrespective of whether a keyring backend is available:
After, if no password is set and there is no keyring backend available, this method now returns None which means that the request runs without auth:
My workaround is to set both the username and password for auth, which is also supported for my case.
Description
Using
poetry installin a context without a keyring backend (in this case a devcontainer), trying to pull packages from a private http-basic secured pypi, causes 401 unauthorized in the install process.poetry lockworks, and certain GETs earlier inpoetry installsucceed as well.devcontainer.json dockerfile.txt
Workarounds
PYTHON_KEYRING_BACKEND=keyring.backends.null.Keyring poetry install -vvvworksPoetry Installation Method
pipx
Operating System
Linux 43d0d0761105 6.5.0-15-generic #15-Ubuntu SMP PREEMPT_DYNAMIC Tue Jan 9 22:39:36 UTC 2024 x86_64 GNU/Linux
Poetry Version
1.8.1, 1.7.1
Poetry Configuration
Example pyproject.toml
Poetry Runtime Logs
later