search

python-poetry / poetry

Python packaging and dependency management made easy
https://python-poetry.org
MIT License
31.08k stars 2.26k forks source link

_add_incompatibility AssertionError with prodigy wheel #6607

Open NixBiks opened 1 year ago

NixBiks commented 1 year ago 
[tool.poetry]
name = "prodigy"
version = "0.1.0"
description = ""
authors = ["Nicolai Bjerre Pedersen <mr.bjerre@gmail.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.9"

[[tool.poetry.source]]
name = "prodigy"
url = "https://download.prodi.gy"
default = false
secondary = false

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Issue

I'm trying to install a package from a private repo but I'm seeing different bugs in the process. First I add the repo by

poetry source add prodigy "https://download.prodi.gy"

Then I add http-basic auth with

poetry config http-basic.prodigy <my-licence-key> ""

And finally I'm trying to add the package with

poetry add --source=prodigy prodigy -vvv

which yields

Loading configuration file /Users/nix/Library/Preferences/pypoetry/config.toml
Loading configuration file /Users/nix/Library/Preferences/pypoetry/auth.toml
Adding repository prodigy (https://download.prodi.gy)
Using virtualenv: /Users/nix/Projects/rnd/prodigy/.venv
Project environment contains an empty path in sys_path, ignoring.
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
[keyring.backend] Loading chainer
[keyring.backend] Loading libsecret
[keyring.backend] Loading macOS
Creating new session for download.prodi.gy
[urllib3.connectionpool] Starting new HTTPS connection (1): download.prodi.gy:443
[urllib3.connectionpool] https://download.prodi.gy:443 "GET /prodigy/ HTTP/1.1" 307 None
[urllib3.connectionpool] https://download.prodi.gy:443 "GET /prodigy HTTP/1.1" 200 716848
Source (prodigy): Response URL https://download.prodi.gy/prodigy differs from request URL https://download.prodi.gy/prodigy/
Source (prodigy): 49 packages found for prodigy *
Using version ^1.11.8 for prodigy

Updating dependencies
Resolving dependencies...
   1: fact: prodigy is 0.1.0
   1: derived: prodigy
   1: Version solving took 0.002 seconds.
   1: Tried 1 solutions.

  Stack trace:

  16  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/cleo/application.py:329 in run
       327│ 
       328│             try:
     → 329│                 exit_code = self._run(io)
       330│             except Exception as e:
       331│                 if not self._catch_exceptions:

  15  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/console/application.py:185 in _run
       183│         self._load_plugins(io)
       184│ 
     → 185│         exit_code: int = super()._run(io)
       186│         return exit_code
       187│ 

  14  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/cleo/application.py:423 in _run
       421│             io.input.set_stream(stream)
       422│ 
     → 423│         exit_code = self._run_command(command, io)
       424│         self._running_command = None
       425│ 

  13  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/cleo/application.py:465 in _run_command
       463│ 
       464│         if error is not None:
     → 465│             raise error
       466│ 
       467│         return event.exit_code

  12  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/cleo/application.py:449 in _run_command
       447│ 
       448│             if event.command_should_run():
     → 449│                 exit_code = command.run(io)
       450│             else:
       451│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

  11  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/cleo/commands/base_command.py:119 in run
       117│         io.input.validate()
       118│ 
     → 119│         status_code = self.execute(io)
       120│ 
       121│         if status_code is None:

  10  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/cleo/commands/command.py:83 in execute
        81│ 
        82│         try:
     →  83│             return self.handle()
        84│         except KeyboardInterrupt:
        85│             return 1

   9  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/console/commands/add.py:255 in handle
       253│         self.installer.whitelist([r["name"] for r in requirements])
       254│ 
     → 255│         status = self.installer.run()
       256│ 
       257│         if status == 0 and not self.option("dry-run"):

   8  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/installer.py:114 in run
       112│             self._execute_operations = False
       113│ 
     → 114│         return self._do_install()
       115│ 
       116│     def dry_run(self, dry_run: bool = True) -> Installer:

   7  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/installation/installer.py:247 in _do_install
       245│                 source_root=self._env.path.joinpath("src")
       246│             ):
     → 247│                 ops = solver.solve(use_latest=self._whitelist).calculate_operations()
       248│         else:
       249│             self._io.write_line("Installing dependencies from lock file")

   6  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/puzzle/solver.py:73 in solve
        71│         with self._provider.progress():
        72│             start = time.time()
     →  73│             packages, depths = self._solve(use_latest=use_latest)
        74│             end = time.time()
        75│ 

   5  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/puzzle/solver.py:151 in _solve
       149│ 
       150│         try:
     → 151│             result = resolve_version(
       152│                 self._package, self._provider, locked=locked, use_latest=use_latest
       153│             )

   4  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/mixology/__init__.py:24 in resolve_version
        22│     solver = VersionSolver(root, provider, locked=locked, use_latest=use_latest)
        23│ 
     →  24│     return solver.solve()
        25│ 

   3  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/mixology/version_solver.py:127 in solve
       125│             while next is not None:
       126│                 self._propagate(next)
     → 127│                 next = self._choose_package_version()
       128│ 
       129│             return self._result()

   2  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/mixology/version_solver.py:450 in _choose_package_version
       448│         conflict = False
       449│         for incompatibility in self._provider.incompatibilities_for(package):
     → 450│             self._add_incompatibility(incompatibility)
       451│ 
       452│             # If an incompatibility is already satisfied, then selecting version

   1  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/mixology/version_solver.py:486 in _add_incompatibility
       484│ 
       485│     def _add_incompatibility(self, incompatibility: Incompatibility) -> None:
     → 486│         self._log(f"fact: {incompatibility}")
       487│ 
       488│         for term in incompatibility.terms:

  AssertionError

  at ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/mixology/incompatibility.py:114 in __str__
      110│         )
      111│ 
      112│     def __str__(self) -> str:
      113│         if isinstance(self._cause, DependencyCause):
    → 114│             assert len(self._terms) == 2
      115│ 
      116│             depender = self._terms[0]
      117│             dependee = self._terms[1]
      118│             assert depender.is_positive()

However the first time I tried I got another error: 403 Client Error: Forbidden for url: https://s3.eu-... so I figured that my licence key was wrong (I later realized that it is most likely related to #4497). So I tried to add it again with poetry config http-basic.prodigy <my-licence-key> "" which then gave this error

Unable to store the password for poetry-repository-prodigy in the key ring: Can't store password on keychain: (-25299, 'Unknown Error')

I checked in auth.toml and it looks fine.... If I add verbose logging on the poetry config command then I get 

Loading configuration file /Users/nix/Library/Preferences/pypoetry/config.toml
Loading configuration file /Users/nix/Library/Preferences/pypoetry/auth.toml
Adding repository prodigy (https://download.prodi.gy)
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
[keyring.backend] Loading chainer
[keyring.backend] Loading libsecret
[keyring.backend] Loading macOS

  Stack trace:

  2  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/keyring/backends/macOS/__init__.py:40 in set_password
       38│ 
       39│         try:
    →  40│             api.set_generic_password(self.keychain, service, username, password)
       41│         except api.KeychainDenied as e:
       42│             raise KeyringLocked("Can't store password on keychain: " "{}".format(e))

  1  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/keyring/backends/macOS/api.py:161 in set_generic_password
      159│ 
      160│     status = SecItemAdd(q, None)
    → 161│     Error.raise_for_status(status)
      162│ 
      163│ 

  Error

  (-25299, 'Unknown Error')

  at ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/keyring/backends/macOS/api.py:114 in raise_for_status
      110│                 status,
      111│                 "Security Auth Failure: make sure "
      112│                 "python is signed with codesign util",
      113│             )
    → 114│         raise cls(status, "Unknown Error")
      115│ 
      116│ 
      117│ class NotFound(Error):
      118│     pass

The following error occurred when trying to handle this error:

  Stack trace:

  2  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/utils/password_manager.py:85 in set_password
       83│ 
       84│         try:
    →  85│             keyring.set_password(name, username, password)
       86│         except (RuntimeError, keyring.errors.KeyringError) as e:
       87│             raise PoetryKeyringError(

  1  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/keyring/core.py:60 in set_password
       58│ def set_password(service_name: str, username: str, password: str) -> None:
       59│     """Set password for the user in the specified service."""
    →  60│     get_keyring().set_password(service_name, username, password)
       61│ 
       62│ 

  PasswordSetError

  Can't store password on keychain: (-25299, 'Unknown Error')

  at ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/keyring/backends/macOS/__init__.py:44 in set_password
       40│             api.set_generic_password(self.keychain, service, username, password)
       41│         except api.KeychainDenied as e:
       42│             raise KeyringLocked("Can't store password on keychain: " "{}".format(e))
       43│         except api.Error as e:
    →  44│             raise PasswordSetError("Can't store password on keychain: " "{}".format(e))
       45│ 
       46│     def get_password(self, service, username):
       47│         if username is None:
       48│             username = ''

The following error occurred when trying to handle this error:

  Stack trace:

  9  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/cleo/application.py:329 in run
      327│ 
      328│             try:
    → 329│                 exit_code = self._run(io)
      330│             except Exception as e:
      331│                 if not self._catch_exceptions:

  8  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/console/application.py:185 in _run
      183│         self._load_plugins(io)
      184│ 
    → 185│         exit_code: int = super()._run(io)
      186│         return exit_code
      187│ 

  7  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/cleo/application.py:423 in _run
      421│             io.input.set_stream(stream)
      422│ 
    → 423│         exit_code = self._run_command(command, io)
      424│         self._running_command = None
      425│ 

  6  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/cleo/application.py:465 in _run_command
      463│ 
      464│         if error is not None:
    → 465│             raise error
      466│ 
      467│         return event.exit_code

  5  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/cleo/application.py:449 in _run_command
      447│ 
      448│             if event.command_should_run():
    → 449│                 exit_code = command.run(io)
      450│             else:
      451│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

  4  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/cleo/commands/base_command.py:119 in run
      117│         io.input.validate()
      118│ 
    → 119│         status_code = self.execute(io)
      120│ 
      121│         if status_code is None:

  3  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/cleo/commands/command.py:83 in execute
       81│ 
       82│         try:
    →  83│             return self.handle()
       84│         except KeyboardInterrupt:
       85│             return 1

  2  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/console/commands/config.py:267 in handle
      265│                     password = values[1]
      266│ 
    → 267│                 password_manager.set_http_password(m.group(2), username, password)
      268│             elif m.group(1) == "pypi-token":
      269│                 if len(values) != 1:

  1  ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/utils/password_manager.py:219 in set_http_password
      217│             auth["password"] = password
      218│         else:
    → 219│             self.keyring.set_password(name, username, password)
      220│ 
      221│         self._config.auth_config_source.add_property(f"http-basic.{name}", auth)

  PoetryKeyringError

  Unable to store the password for poetry-repository-prodigy in the key ring: Can't store password on keychain: (-25299, 'Unknown Error')

  at ~/Library/Application Support/pypoetry/venv/lib/python3.10/site-packages/poetry/utils/password_manager.py:87 in set_password
       83│ 
       84│         try:
       85│             keyring.set_password(name, username, password)
       86│         except (RuntimeError, keyring.errors.KeyringError) as e:
    →  87│             raise PoetryKeyringError(
       88│                 f"Unable to store the password for {name} in the key ring: {e}"
       89│             )
       90│ 
       91│     def delete_password(self, name: str, username: str) -> None:

I know this issue is getting out of hand but I'm not sure if these errors are all linked to eachother?

There is also an ongoing thread here.

neersighted commented 1 year ago

The keyring error is based on the fact the macOS keyring doesn't let you store an internet password as a username only. Despite the fact we were unable to set things in the keyring/threw an exception, we fell back to setting auth.toml successfully.

What is more interesting is that the solver crashes on an assertion -- either our typing is incorrect, or prodigy wheels are malformed.

I suspect @radoering and @dimbleby will be interested in looking at this. The Poetry team will need access to these wheels to introspect them. You can provide them privately by emailing me (you may find my email in Git history) and I can distribute them to the relevant parties -- alternatively, have the Prodigy people reach out via the same method.

neersighted commented 1 year ago

Also, please provide a sanitized pyproject.toml per the issue template -- thanks!

NixBiks commented 1 year ago

Also, please provide a sanitized pyproject.toml per the issue template -- thanks!

@neersighted I thought I did already? (or was it just the sanitized bit - marking it as toml?)

neersighted commented 1 year ago

I wasn't sure if your minimal pyproject.toml reproduced with only prodigy, or if there were more dependencies. Typically we want to see the actual pyproject.toml (or as close as we can get) when there are solver errors. However in this case it looks like just prodigy might be enough?

NixBiks commented 1 year ago

Yes the minimal example is enough - I can't get to 403 Client Error: Forbidden for url: https://s3.eu-... anymore though. That only happened once on my first attempt

neersighted commented 1 year ago

It would have happened once due to not having credentials set up properly -- it should not reproduce now that they're in auth.toml.

NixBiks commented 1 year ago

I'm not so sure actually. Since the credentials is needed to get the wheel link - see #4497.

The error occurs due to poetry passing my HTTP Basic auth credentials to the AWS S3 pre-signed URL. AWS throws the error saying there can only be 1 auth method (which is the pre-signed URL) but Basic Auth credentials were also passed

So there might be multiple problems here...

neersighted commented 1 year ago

I don't think that is at all what is going on -- that wouldn't result in a 403, and the authenticator code has been pretty much replaced. Poetry is clearly able to grab the wheel given it can count releases and chokes on solving for incompatibility.

NixBiks commented 1 year ago

The keyring error is based on the fact the macOS keyring doesn't let you store an internet password as a username only.

Btw. I get the same error if I try to add a password as well, i.e. Unable to store the password for poetry-repository-prodigy in the key ring: Can't store password on keychain: (-25299, 'Unknown Error')

neersighted commented 1 year ago

I'm not sure why your keyring is broken, but you can isolate it by turning off support and forcing the auth.toml fallback: see #1917 for methods to do so.

NixBiks commented 1 year ago

Ah so I can go to Keychain Access in macOS and delete the keyring from there. Then I can add it again.

Now it becomes really strange though. Instead of doing poetry config http-basic.prodigy <my-licence-key> "" I did

poetry config http-basic.prodigy <my-licence-key>

and then just clicked enter twice to confirm an empty password. After this I was able to install prodigy in my main project!?

However I still can't install prodigy this way in a fresh project. Very strange

neersighted commented 1 year ago

I suggest we focus on the solver issue here, since once we solve that it's easier to isolate and open issues for other problems. When reproducing, I'd always suggest using a container to isolate local configuration issues -- e.g. can you try to solve with a fresh container?

NixBiks commented 1 year ago

Sounds like a good idea!

A docker container? Do you have an image template you use or?

neersighted commented 1 year ago

The python:3.x library images on the Docker Hub are great as they use Python.org builds with no patches, and include most of the software you want (e.g. git, a C compiler, etc) out of the box.

NixBiks commented 1 year ago

So I made a container with

FROM python:3.9.9
ENV POETRY_HOME=/usr/local
RUN curl -sSL https://install.python-poetry.org | python3 -

Then I ran a interactive session and did the following

mkdir project && cd project
poetry init
poetry source add prodigy "https://download.prodi.gy"
poetry config http-basic.prodigy <my-licence-key>
poetry add --source prodigy prodigy

And it just worked. So I'm not sure what's going on in my other case there...

When I added the auth I got this

> poetry config http-basic.prodigy <my-licence>
Password: 

Using a plaintext file to store credentials

Invalid type <class 'NoneType'>

but it worked never the less

mrcontre commented 1 year ago

Hi @NixBiks , did you ever figure out why the Invalid type <class 'NoneType'> appeared? I'm running into some some issues with in builds because of it.

samuller commented 1 year ago

@mrcontre If you see the detailed error by adding -vvv to the command (e.g. poetry -vvv config ...), then it shows the error Invalid type <class 'NoneType'> itself comes from the tomlkit library (see related issue), but the mistake is actually much simpler - it happens when you try set the password config with an empty value.

I had the exact same issue in CI only, but it turns out it was because the environment variable wasn't set, e.g. when running poetry config http-basic.local_registry username $REGISTRY_PASSWORD and REGISTRY_PASSWORD wasn't set.

The error message could definitely be more helpful, although this seems to be a bit of an edge-case because if there's an interactive terminal then the command would try and allow the password to be entered via prompt (you can duplicate the issue then by just pressing enter twice at that prompt).

mrcontre commented 1 year ago

Thanks @samuller, I eventually found the issue and if I'm not mistaken it had something to do with an unset environment variable as you pointed out. A more specific error message would definitely help!