[Alpha/Beta] Cannot pull package from Google Artifact Registry with Keyring Auth

[Misterturtle]

• [ ] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: MacOS Catalina Version 10.15.7
• Poetry version: 1.2.0b1
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/Misterturtle/a8d5f4485960bdcec07dd45a1f543ab2

Issue

Dependencies:

Artifact Registry Keyring: pip list | grep keyring

keyring                               23.5.0
keyrings.google-artifactregistry-auth 1.0.0

Python Version: python --version

Python 3.9.4

Poetry Version poetry --version

Poetry (version 1.2.0b1)

Gcloud config: gcloud config list

[compute]
region = us-central1
zone = us-central1-c
[core]
account = ryan@conaway58.com
disable_usage_reporting = False
project = digital-ryan

Your active configuration is: [default]

Minimum Working Example: When running poetry add poetry-test -vvv, poetry is unable to download and install a package from GCP Artifact Registry.

• Tried with both Poetry 1.2.0b1 and Poetry 1.2.0a2
• Keyring is installed
• gcloud is configured
• The python package is accessible from my current network with pip command

Loading configuration file /Users/my_user/Library/Application Support/pypoetry/config.toml
Loading configuration file /Users/my_user/Projects/poetry-test/poetry.toml
Adding repository poetest (https://us-central1-pypi.pkg.dev/digital-ryan/poetest/simple) and setting it as secondary
Using virtualenv: /Users/my_user/Projects/poetry-test/.venv
PyPI: No packages found for poetry-test *

  Stack trace:

  9  ~/.pyenv/versions/3.9.4/lib/python3.9/site-packages/cleo/application.py:330 in run
      328│ 
      329│             try:
    → 330│                 exit_code = self._run(io)
      331│             except Exception as e:
      332│                 if not self._catch_exceptions:

  8  ~/.pyenv/versions/3.9.4/lib/python3.9/site-packages/poetry/console/application.py:174 in _run
      172│         self._load_plugins(io)
      173│ 
    → 174│         return super()._run(io)
      175│ 
      176│     def _configure_io(self, io: IO) -> None:

  7  ~/.pyenv/versions/3.9.4/lib/python3.9/site-packages/cleo/application.py:425 in _run
      423│                 io.set_input(ArgvInput(argv))
      424│ 
    → 425│         exit_code = self._run_command(command, io)
      426│         self._running_command = None
      427│ 

  6  ~/.pyenv/versions/3.9.4/lib/python3.9/site-packages/cleo/application.py:467 in _run_command
      465│ 
      466│         if error is not None:
    → 467│             raise error
      468│ 
      469│         return event.exit_code

  5  ~/.pyenv/versions/3.9.4/lib/python3.9/site-packages/cleo/application.py:451 in _run_command
      449│ 
      450│             if event.command_should_run():
    → 451│                 exit_code = command.run(io)
      452│             else:
      453│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

  4  ~/.pyenv/versions/3.9.4/lib/python3.9/site-packages/cleo/commands/base_command.py:118 in run
      116│         io.input.validate()
      117│ 
    → 118│         status_code = self.execute(io)
      119│ 
      120│         if status_code is None:

  3  ~/.pyenv/versions/3.9.4/lib/python3.9/site-packages/cleo/commands/command.py:85 in execute
       83│ 
       84│         try:
    →  85│             return self.handle()
       86│         except KeyboardInterrupt:
       87│             return 1

  2  ~/.pyenv/versions/3.9.4/lib/python3.9/site-packages/poetry/console/commands/add.py:148 in handle
      146│             return 0
      147│ 
    → 148│         requirements = self._determine_requirements(
      149│             packages,
      150│             allow_prereleases=self.option("allow-prereleases"),

  1  ~/.pyenv/versions/3.9.4/lib/python3.9/site-packages/poetry/console/commands/init.py:357 in _determine_requirements
      355│             elif "version" not in requirement:
      356│                 # determine the best version automatically
    → 357│                 name, version = self._find_best_version_for_package(
      358│                     requirement["name"],
      359│                     allow_prereleases=allow_prereleases,

  ValueError

  Could not find a matching version of package poetry-test

  at ~/.pyenv/versions/3.9.4/lib/python3.9/site-packages/poetry/console/commands/init.py:398 in _find_best_version_for_package
      394│         )
      395│ 
      396│         if not package:
      397│             # TODO: find similar
    → 398│             raise ValueError(f"Could not find a matching version of package {name}")
      399│ 
      400│         return package.pretty_name, selector.find_recommended_require_version(package)
      401│ 
      402│     def _parse_requirements(self, requirements: list[str]) -> list[dict[str, Any]]:

Proving the Artifact Registry image exists and is accessible:

Installing the same package through pip + keyring successfully downloads and installs the poetry-test package

poetry run pip install --extra-index-url https://us-central1-python.pkg.dev/digital-ryan/poetest/simple poetry-test -vvv

Using pip 21.0.1 from /Users/my_user/Projects/poetry-test/.venv/lib/python3.9/site-packages/pip (python 3.9)
Non-user install because user site-packages disabled
Created temporary directory: /private/var/folders/6t/0nckgp4x26q8xj3v6s2234d00000gn/T/pip-ephem-wheel-cache-_f764u0k
Created temporary directory: /private/var/folders/6t/0nckgp4x26q8xj3v6s2234d00000gn/T/pip-req-tracker-j2ui8c81
Initialized build tracking at /private/var/folders/6t/0nckgp4x26q8xj3v6s2234d00000gn/T/pip-req-tracker-j2ui8c81
Created build tracker: /private/var/folders/6t/0nckgp4x26q8xj3v6s2234d00000gn/T/pip-req-tracker-j2ui8c81
Entered build tracker: /private/var/folders/6t/0nckgp4x26q8xj3v6s2234d00000gn/T/pip-req-tracker-j2ui8c81
Created temporary directory: /private/var/folders/6t/0nckgp4x26q8xj3v6s2234d00000gn/T/pip-install-4ijqeh2c
Looking in indexes: https://pypi.org/simple, https://us-central1-python.pkg.dev/digital-ryan/poetest/simple
2 location(s) to search for versions of poetry-test:
* https://pypi.org/simple/poetry-test/
* https://us-central1-python.pkg.dev/digital-ryan/poetest/simple/poetry-test/
Fetching project page and analyzing links: https://pypi.org/simple/poetry-test/
Getting page https://pypi.org/simple/poetry-test/
Found index url https://pypi.org/simple
Getting credentials from keyring for https://pypi.org/simple
Loading Google Auth
Loading KWallet
Loading SecretService
Loading Windows
Loading chainer
Loading libsecret
Loading macOS
Getting credentials from keyring for pypi.org
Looking up "https://pypi.org/simple/poetry-test/" in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/poetry-test/ HTTP/1.1" 404 13
Status code 404 not in (200, 203, 300, 301)
Could not fetch URL https://pypi.org/simple/poetry-test/: 404 Client Error: Not Found for url: https://pypi.org/simple/poetry-test/ - skipping
Fetching project page and analyzing links: https://us-central1-python.pkg.dev/digital-ryan/poetest/simple/poetry-test/
Getting page https://us-central1-python.pkg.dev/digital-ryan/poetest/simple/poetry-test/
Found index url https://us-central1-python.pkg.dev/digital-ryan/poetest/simple
Getting credentials from keyring for https://us-central1-python.pkg.dev/digital-ryan/poetest/simple
Checking None for explicit credentials as part of auth process...
Checking Cloud SDK credentials as part of auth process...
Making request: POST https://oauth2.googleapis.com/token
Starting new HTTPS connection (1): oauth2.googleapis.com:443
https://oauth2.googleapis.com:443 "POST /token HTTP/1.1" 200 None
Found credentials in keyring for us-central1-python.pkg.dev
Looking up "https://us-central1-python.pkg.dev/digital-ryan/poetest/simple/poetry-test/" in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTPS connection (1): us-central1-python.pkg.dev:443
https://us-central1-python.pkg.dev:443 "GET /digital-ryan/poetest/simple/poetry-test/ HTTP/1.1" 200 444
Updating cache with response from "https://us-central1-python.pkg.dev/digital-ryan/poetest/simple/poetry-test/"
  Found link https://us-central1-python.pkg.dev/digital-ryan/poetest/poetry-test/poetry-test-0.1.7.tar.gz (from https://us-central1-python.pkg.dev/digital-ryan/poetest/simple/poetry-test/), version: 0.1.7
  Found link https://us-central1-python.pkg.dev/digital-ryan/poetest/poetry-test/poetry_test-0.1.7-py3-none-any.whl (from https://us-central1-python.pkg.dev/digital-ryan/poetest/simple/poetry-test/), version: 0.1.7
Given no hashes to check 2 links for project 'poetry-test': discarding no candidates
Collecting poetry-test
  Created temporary directory: /private/var/folders/6t/0nckgp4x26q8xj3v6s2234d00000gn/T/pip-unpack-0hv5e8_r
  Looking up "https://us-central1-python.pkg.dev/digital-ryan/poetest/poetry-test/poetry_test-0.1.7-py3-none-any.whl" in the cache
  No cache entry available
  https://us-central1-python.pkg.dev:443 "GET /digital-ryan/poetest/poetry-test/poetry_test-0.1.7-py3-none-any.whl HTTP/1.1" 307 0
  Status code 307 not in (200, 203, 300, 301)
  Looking up "https://us-central1-python.pkg.dev/artifacts-downloads/namespaces/digital-ryan/repositories/poetest/downloads/AHf3wE4waeHGwn8CLy0iNGHYoKpewIa7dKhWDTV_s6Os5Lizm6PcGTT1z_BybPbTJSAMTTKkDiJ3RQqdIGRPNuknILipoSJs55rgpoemqDoPAQtSQN_IgnnfaHuIkqodPw-glhRpRTv4SyAi5slDObOmHRh6Jlqtnjbalt8dcwj4h_tI9PJyZcUyHtmC-X4gh_Q6KNB-2S52MPKaSFqGJ5H6KP3gNTaZ3_Sc4poIUeLDwD3z0MmG0Fqw3NBu8UdfWDf5LU19WVV6nY_dXqU-FR-1qVjIyjyFvOP6JiFPS8qRXH5d4PgshPrkJ8LZnAPmvM06kPHp_51yqcBM31b1LSZFnTc1O04eEq_14cuvnYDy2LG607JDNCU5WVM77vBzKED-6ZXwNzeetz6KTqhlAgWm2QYf5BKM5Ul4FBqg-GwFZz03CmzSdcfBESwrqgOrfKRx_pWmteDabXFF92_5yL0xGuI1_4bDNWl4WjWMnXSVS2xTQsCCLS_Oi6Sd_wJhRiJA7TP0JilubdDnVZp_HsRGqyaNW1ZuvWH-JqxGELyDkYB4CXd8J4DpzyyPEzXO82DYuH0Kqyq_J5zH2i1Iadpx2tywKgTjX192gam-MnY1Y1aARS-HLcXl1YLvmTRwzwfEigVADAbaarb0uQFc5_o11cBIrwEQ8pMeKCKALPM0qAe32nHVSXA4mHYkWA9aUKrQ5Aa4F_xnxFfPjnVoKHShaluroytZhROZY0bB8kov27RW87rc2HuyXIz0zPG3dCl4_47U5gTgl8zQF-MkYirh0Zzj5zcZl_0nCWBb5XvNHYGKgRvpUgFJbQ-VwRykd1JUkQZeigc_BQUzaaXrvlpC5qs3GqTLrdy4HddRgghmZncEk1S23ve5ljnF9K3tlAjQbN3NcDhnoBJ9RLaYXVtf4xHCK9hk_fxSbLVcGjlMJkckT8-hNWaQiUGJy7d0VlzWUKmrSVastKCVkMRx42fozm7wj0U7RCRcO-oEXAl29msgPMVbpzBLD-tlo-Sf7l_zPGQ=" in the cache
  No cache entry available
  https://us-central1-python.pkg.dev:443 "GET /artifacts-downloads/namespaces/digital-ryan/repositories/poetest/downloads/AHf3wE4waeHGwn8CLy0iNGHYoKpewIa7dKhWDTV_s6Os5Lizm6PcGTT1z_BybPbTJSAMTTKkDiJ3RQqdIGRPNuknILipoSJs55rgpoemqDoPAQtSQN_IgnnfaHuIkqodPw-glhRpRTv4SyAi5slDObOmHRh6Jlqtnjbalt8dcwj4h_tI9PJyZcUyHtmC-X4gh_Q6KNB-2S52MPKaSFqGJ5H6KP3gNTaZ3_Sc4poIUeLDwD3z0MmG0Fqw3NBu8UdfWDf5LU19WVV6nY_dXqU-FR-1qVjIyjyFvOP6JiFPS8qRXH5d4PgshPrkJ8LZnAPmvM06kPHp_51yqcBM31b1LSZFnTc1O04eEq_14cuvnYDy2LG607JDNCU5WVM77vBzKED-6ZXwNzeetz6KTqhlAgWm2QYf5BKM5Ul4FBqg-GwFZz03CmzSdcfBESwrqgOrfKRx_pWmteDabXFF92_5yL0xGuI1_4bDNWl4WjWMnXSVS2xTQsCCLS_Oi6Sd_wJhRiJA7TP0JilubdDnVZp_HsRGqyaNW1ZuvWH-JqxGELyDkYB4CXd8J4DpzyyPEzXO82DYuH0Kqyq_J5zH2i1Iadpx2tywKgTjX192gam-MnY1Y1aARS-HLcXl1YLvmTRwzwfEigVADAbaarb0uQFc5_o11cBIrwEQ8pMeKCKALPM0qAe32nHVSXA4mHYkWA9aUKrQ5Aa4F_xnxFfPjnVoKHShaluroytZhROZY0bB8kov27RW87rc2HuyXIz0zPG3dCl4_47U5gTgl8zQF-MkYirh0Zzj5zcZl_0nCWBb5XvNHYGKgRvpUgFJbQ-VwRykd1JUkQZeigc_BQUzaaXrvlpC5qs3GqTLrdy4HddRgghmZncEk1S23ve5ljnF9K3tlAjQbN3NcDhnoBJ9RLaYXVtf4xHCK9hk_fxSbLVcGjlMJkckT8-hNWaQiUGJy7d0VlzWUKmrSVastKCVkMRx42fozm7wj0U7RCRcO-oEXAl29msgPMVbpzBLD-tlo-Sf7l_zPGQ= HTTP/1.1" 200 11797
  Downloading https://us-central1-python.pkg.dev/digital-ryan/poetest/poetry-test/poetry_test-0.1.7-py3-none-any.whl (11 kB)
  Updating cache with response from "https://us-central1-python.pkg.dev/artifacts-downloads/namespaces/digital-ryan/repositories/poetest/downloads/AHf3wE4waeHGwn8CLy0iNGHYoKpewIa7dKhWDTV_s6Os5Lizm6PcGTT1z_BybPbTJSAMTTKkDiJ3RQqdIGRPNuknILipoSJs55rgpoemqDoPAQtSQN_IgnnfaHuIkqodPw-glhRpRTv4SyAi5slDObOmHRh6Jlqtnjbalt8dcwj4h_tI9PJyZcUyHtmC-X4gh_Q6KNB-2S52MPKaSFqGJ5H6KP3gNTaZ3_Sc4poIUeLDwD3z0MmG0Fqw3NBu8UdfWDf5LU19WVV6nY_dXqU-FR-1qVjIyjyFvOP6JiFPS8qRXH5d4PgshPrkJ8LZnAPmvM06kPHp_51yqcBM31b1LSZFnTc1O04eEq_14cuvnYDy2LG607JDNCU5WVM77vBzKED-6ZXwNzeetz6KTqhlAgWm2QYf5BKM5Ul4FBqg-GwFZz03CmzSdcfBESwrqgOrfKRx_pWmteDabXFF92_5yL0xGuI1_4bDNWl4WjWMnXSVS2xTQsCCLS_Oi6Sd_wJhRiJA7TP0JilubdDnVZp_HsRGqyaNW1ZuvWH-JqxGELyDkYB4CXd8J4DpzyyPEzXO82DYuH0Kqyq_J5zH2i1Iadpx2tywKgTjX192gam-MnY1Y1aARS-HLcXl1YLvmTRwzwfEigVADAbaarb0uQFc5_o11cBIrwEQ8pMeKCKALPM0qAe32nHVSXA4mHYkWA9aUKrQ5Aa4F_xnxFfPjnVoKHShaluroytZhROZY0bB8kov27RW87rc2HuyXIz0zPG3dCl4_47U5gTgl8zQF-MkYirh0Zzj5zcZl_0nCWBb5XvNHYGKgRvpUgFJbQ-VwRykd1JUkQZeigc_BQUzaaXrvlpC5qs3GqTLrdy4HddRgghmZncEk1S23ve5ljnF9K3tlAjQbN3NcDhnoBJ9RLaYXVtf4xHCK9hk_fxSbLVcGjlMJkckT8-hNWaQiUGJy7d0VlzWUKmrSVastKCVkMRx42fozm7wj0U7RCRcO-oEXAl29msgPMVbpzBLD-tlo-Sf7l_zPGQ="
  Caching b/c of expires header
  Added poetry-test from https://us-central1-python.pkg.dev/digital-ryan/poetest/poetry-test/poetry_test-0.1.7-py3-none-any.whl to build tracker '/private/var/folders/6t/0nckgp4x26q8xj3v6s2234d00000gn/T/pip-req-tracker-j2ui8c81'
  Removed poetry-test from https://us-central1-python.pkg.dev/digital-ryan/poetest/poetry-test/poetry_test-0.1.7-py3-none-any.whl from build tracker '/private/var/folders/6t/0nckgp4x26q8xj3v6s2234d00000gn/T/pip-req-tracker-j2ui8c81'
Requirement already satisfied: pydantic<2.0.0,>=1.9.0 in ./.venv/lib/python3.9/site-packages (from poetry-test) (1.9.0)
Requirement already satisfied: typer<0.5.0,>=0.4.0 in ./.venv/lib/python3.9/site-packages (from poetry-test) (0.4.0)
Requirement already satisfied: PyYAML<6.0.0,>=5.4.1 in ./.venv/lib/python3.9/site-packages (from poetry-test) (5.4.1)
Requirement already satisfied: typing-extensions>=3.7.4.3 in ./.venv/lib/python3.9/site-packages (from pydantic<2.0.0,>=1.9.0->poetry-test) (4.1.1)
Requirement already satisfied: click<9.0.0,>=7.1.1 in ./.venv/lib/python3.9/site-packages (from typer<0.5.0,>=0.4.0->poetry-test) (8.0.4)
Created temporary directory: /private/var/folders/6t/0nckgp4x26q8xj3v6s2234d00000gn/T/pip-unpack-6aficrfk
Installing collected packages: poetry-test

  changing mode of /Users/my_user/Projects/poetry-test/.venv/bin/mach1ml to 755
Successfully installed poetry-test-0.1.7
WARNING: You are using pip version 21.0.1; however, version 22.0.4 is available.
You should consider upgrading via the '/Users/my_user/Projects/poetry-test/.venv/bin/python -m pip install --upgrade pip' command.
Removed build tracker: '/private/var/folders/6t/0nckgp4x26q8xj3v6s2234d00000gn/T/pip-req-tracker-j2ui8c81'

[Darsstar]

How did you install Poetry?

If via pipx try running pipx inject poetry keyrings.google-artifactregistry-auth.

If via install-poetry.py/get-poetry.py try running poetry plugin add keyrings.google-artifactregistry-auth

If via pip either the latter or pip should do, but I doubt it would have an effect.

If you did not use pip, or if you manually created a virtual environment, poetry runs in it's own venv and does not call pip to download wheels, so the keyring backend needs to be available in it's venv. poetry plugin add is the way poetry provides for doing that. Pipx also has it's own pipx inject to achieve that.

[Misterturtle]

I installed poetry directly with pip, which is being managed by pyenv: pip install --upgrade poetry==1.2.0b1

$ which pip
/Users/my_user/.pyenv/versions/3.9.4/bin/pip

$ pyenv global
3.9.4

I successfully ran your suggestion for adding it as a plugin, but the error still occurs: poetry plugin add keyrings.google-artifactregistry-auth

Furthermore, if I uninstall the keyring and try to pull a package, it asks for username/password, as opposed to when the keyring is installed, it successfully retrieves the credentials without input. Also, I am able to publish packages to Artifact Registry with the keyring. I don't believe its directly a keyring issue.

Also, for further context (albeit anecdotal), my team has been using Poetry Alpha + Keyring +Art Reg for months successfully, but suddenly we've seen all developers hitting the same issue.

Edit: Here is my keyring configuration inside the poetry venv

$ poetry run keyring --list-backends
keyring.backends.macOS.Keyring (priority: 5)
keyring.backends.fail.Keyring (priority: 0)
keyring.backends.chainer.ChainerBackend (priority: 10)
keyrings.gauth.GooglePythonAuth (priority: 9)

[Misterturtle]

I've found the solution. The issue isn't with poetry, but rather with a breaking change with Artifact Registry. Based on the release notes found here: https://cloud.google.com/artifact-registry/docs/release-notes#March_01_2022

The solution is simply to update pyproject.toml with the host url as described in the release notes. The old hostname format still works via Pip, but not Poetry.

For example:

[[tool.poetry.source]]
name = "poetest"
url = "https://us-central1-pypi.pkg.dev/digital-ryan/poetest/simple/"
default = true

Becomes:

[[tool.poetry.source]]
name = "poetest"
url = "https://us-central1-python.pkg.dev/digital-ryan/poetest/simple/"
default = true

This issue can be closed, but better logging around a failure to connect would have been helpful.

[github-actions[bot]]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.