poetry install fails with AssertionError

eaaltonen commented 6 months ago

Description

Trying to install project with dependency to /vsphere-automation-sdk-python fails with AssertionError.

On an otherwise empty directory with the pyproject.toml provided below, the poetry install command failed.

Workarounds

Use venv and pip instead.

Poetry Installation Method

pip

Operating System

Ubuntu 22.04

Poetry Version

Poetry (version 1.8.2)

Poetry Configuration

cache-dir = "/home/USER/.cache/pypoetry"
experimental.system-git-client = false
installer.max-workers = null
installer.modern-installation = true
installer.no-binary = null
installer.parallel = true
keyring.enabled = true
solver.lazy-wheel = true
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs"  # /home/USER/.cache/pypoetry/virtualenvs
virtualenvs.prefer-active-python = false
virtualenvs.prompt = "{project_name}-py{python_version}"
warnings.export = true

Python Sysconfig

No response

Example pyproject.toml

[tool.poetry]
name = "vmware-py"
version = "0.1.0"
description = "Maintenance scripts for VMware vSphere"
readme = "README.md"
package-mode = false

[tool.poetry.dependencies]
python = "^3.10"
vsphere-automation-sdk = {git = "https://github.com/vmware/vsphere-automation-sdk-python.git"}

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Poetry Runtime Logs

Using virtualenv: /home/USER/.cache/pypoetry/virtualenvs/vmware-py-Z3euOynY-py3.10
Updating dependencies
Resolving dependencies...
   1: fact: vmware-py is 0.1.0
   1: derived: vmware-py
Checking if keyring is available
[keyring:keyring.backend] Loading KWallet
[keyring:keyring.backend] Loading SecretService
[keyring:keyring.backend] Loading Windows
[keyring:keyring.backend] Loading chainer
[keyring:keyring.backend] Loading libsecret
[keyring:keyring.backend] Loading macOS
Using keyring backend 'SecretService Keyring'
[urllib3:urllib3.connectionpool] Starting new HTTPS connection (1): github.com:443
[urllib3:urllib3.connectionpool] https://github.com:443 "GET /vmware/vsphere-automation-sdk-python.git/info/refs?service=git-upload-pack HTTP/1.1" 200 None
Cloning https://github.com/vmware/vsphere-automation-sdk-python.git at 'HEAD' to /home/USER/.cache/pypoetry/virtualenvs/vmware-py-Z3euOynY-py3.10/src/vsphere-automation-sdk-python
   1: fact: vmware-py depends on vsphere-automation-sdk (1.86.0)
   1: selecting vmware-py (0.1.0)
   1: derived: vsphere-automation-sdk (1.86.0) @ git+https://github.com/vmware/vsphere-automation-sdk-python.git
   1: fact: vsphere-automation-sdk (1.86.0) depends on lxml (>=4.3.0)
   1: fact: vsphere-automation-sdk (1.86.0) depends on pyVmomi (>=6.7)
   1: fact: vsphere-automation-sdk (1.86.0) depends on vapi_runtime (2.44.0)
   1: fact: vsphere-automation-sdk (1.86.0) depends on vcenter_bindings (4.2.0)
   1: fact: vsphere-automation-sdk (1.86.0) depends on vapi_common_client (2.44.0)
   1: fact: vsphere-automation-sdk (1.86.0) depends on vmwarecloud_aws (1.64.0)
   1: fact: vsphere-automation-sdk (1.86.0) depends on nsx_python_sdk (4.1.2.0.0)
   1: fact: vsphere-automation-sdk (1.86.0) depends on nsx_policy_python_sdk (4.1.2.0.0)
   1: fact: vsphere-automation-sdk (1.86.0) depends on nsx_vmc_policy_python_sdk (4.1.2.0.0)
   1: fact: vsphere-automation-sdk (1.86.0) depends on nsx_vmc_aws_integration_python_sdk (4.1.2.0.0)
   1: fact: vsphere-automation-sdk (1.86.0) depends on vmwarecloud_draas (1.23.0)
   1: selecting vsphere-automation-sdk (1.86.0 99ee630)
   1: derived: vmwarecloud_draas (1.23.0) @ file:////home/USER/.cache/pypoetry/virtualenvs/vmware-py-Z3euOynY-py3.10/src/vsphere-automation-sdk-python/lib/vmwarecloud-draas/vmwarecloud_draas-1.23.0-py2.py3-none-any.whl
   1: derived: nsx_vmc_aws_integration_python_sdk (4.1.2.0.0) @ file:////home/USER/.cache/pypoetry/virtualenvs/vmware-py-Z3euOynY-py3.10/src/vsphere-automation-sdk-python/lib/nsx-vmc-aws-integration-python-sdk/nsx_vmc_aws_integration_python_sdk-4.1.2.0.0-py2.py3-none-any.whl
   1: derived: nsx_vmc_policy_python_sdk (4.1.2.0.0) @ file:////home/USER/.cache/pypoetry/virtualenvs/vmware-py-Z3euOynY-py3.10/src/vsphere-automation-sdk-python/lib/nsx-vmc-policy-python-sdk/nsx_vmc_policy_python_sdk-4.1.2.0.0-py2.py3-none-any.whl
   1: derived: nsx_policy_python_sdk (4.1.2.0.0) @ file:////home/USER/.cache/pypoetry/virtualenvs/vmware-py-Z3euOynY-py3.10/src/vsphere-automation-sdk-python/lib/nsx-policy-python-sdk/nsx_policy_python_sdk-4.1.2.0.0-py2.py3-none-any.whl
   1: derived: nsx_python_sdk (4.1.2.0.0) @ file:////home/USER/.cache/pypoetry/virtualenvs/vmware-py-Z3euOynY-py3.10/src/vsphere-automation-sdk-python/lib/nsx-python-sdk/nsx_python_sdk-4.1.2.0.0-py2.py3-none-any.whl
   1: derived: vmwarecloud_aws (1.64.0) @ file:////home/USER/.cache/pypoetry/virtualenvs/vmware-py-Z3euOynY-py3.10/src/vsphere-automation-sdk-python/lib/vmwarecloud-aws/vmwarecloud_aws-1.64.0-py2.py3-none-any.whl
   1: derived: vapi_common_client (2.44.0) @ file:////home/USER/.cache/pypoetry/virtualenvs/vmware-py-Z3euOynY-py3.10/src/vsphere-automation-sdk-python/lib/vapi-common-client/vapi_common_client-2.44.0-py2.py3-none-any.whl
   1: derived: vcenter_bindings (4.2.0) @ file:////home/USER/.cache/pypoetry/virtualenvs/vmware-py-Z3euOynY-py3.10/src/vsphere-automation-sdk-python/lib/vcenter-bindings/vcenter_bindings-4.2.0-py2.py3-none-any.whl
   1: derived: vapi_runtime (2.44.0) @ file:////home/USER/.cache/pypoetry/virtualenvs/vmware-py-Z3euOynY-py3.10/src/vsphere-automation-sdk-python/lib/vapi-runtime/vapi_runtime-2.44.0-py2.py3-none-any.whl
   1: derived: pyVmomi (>=6.7)
   1: derived: lxml (>=4.3.0)
Checking if keyring is available
Using keyring backend 'SecretService Keyring'
Creating new session for pypi.org
Skipping url (https://files.pythonhosted.org/packages/d0/ad/f9993f77268d722eea5cac447fc93bcdcf712dc913d607b0fe7eb8449044/pyvmomi-5.5.0-2014.1.tar.gz) due to invalid version (None)
Skipping url (https://files.pythonhosted.org/packages/4d/5c/db408a3529a368c432d58a6b4d1684aa79946d85c5e4e9b8167df08484b4/pyvmomi-5.5.0-2014.1.1.tar.gz) due to invalid version (None)
Source (PyPI): 16 packages found for pyvmomi >=6.7
Source (PyPI): 28 packages found for lxml >=4.3.0
   1: fact: vmwarecloud-draas (1.23.0) depends on setuptools (*)
   1: fact: vmwarecloud-draas (1.23.0) depends on vapi-common-client (*)
   1: fact: vmwarecloud-draas (1.23.0) depends on vapi-runtime (*)
   1: Version solving took 0.811 seconds.
   1: Tried 1 solutions.

  Stack trace:

  17  ~/venvs/poetry/lib/python3.10/site-packages/cleo/application.py:327 in run
       325│ 
       326│             try:
     → 327│                 exit_code = self._run(io)
       328│             except BrokenPipeError:
       329│                 # If we are piped to another process, it may close early and send a

  16  ~/venvs/poetry/lib/python3.10/site-packages/poetry/console/application.py:190 in _run
       188│         self._load_plugins(io)
       189│ 
     → 190│         exit_code: int = super()._run(io)
       191│         return exit_code
       192│ 

  15  ~/venvs/poetry/lib/python3.10/site-packages/cleo/application.py:431 in _run
       429│             io.input.interactive(interactive)
       430│ 
     → 431│         exit_code = self._run_command(command, io)
       432│         self._running_command = None
       433│ 

  14  ~/venvs/poetry/lib/python3.10/site-packages/cleo/application.py:473 in _run_command
       471│ 
       472│         if error is not None:
     → 473│             raise error
       474│ 
       475│         return terminate_event.exit_code

  13  ~/venvs/poetry/lib/python3.10/site-packages/cleo/application.py:457 in _run_command
       455│ 
       456│             if command_event.command_should_run():
     → 457│                 exit_code = command.run(io)
       458│             else:
       459│                 exit_code = ConsoleCommandEvent.RETURN_CODE_DISABLED

  12  ~/venvs/poetry/lib/python3.10/site-packages/cleo/commands/base_command.py:117 in run
       115│         io.input.validate()
       116│ 
     → 117│         return self.execute(io) or 0
       118│ 
       119│     def merge_application_definition(self, merge_args: bool = True) -> None:

  11  ~/venvs/poetry/lib/python3.10/site-packages/cleo/commands/command.py:61 in execute
        59│ 
        60│         try:
     →  61│             return self.handle()
        62│         except KeyboardInterrupt:
        63│             return 1

  10  ~/venvs/poetry/lib/python3.10/site-packages/poetry/console/commands/install.py:153 in handle
       151│         self.installer.verbose(self.io.is_verbose())
       152│ 
     → 153│         return_code = self.installer.run()
       154│ 
       155│         if return_code != 0:

   9  ~/venvs/poetry/lib/python3.10/site-packages/poetry/installation/installer.py:104 in run
       102│             self.verbose(True)
       103│ 
     → 104│         return self._do_install()
       105│ 
       106│     def dry_run(self, dry_run: bool = True) -> Installer:

   8  ~/venvs/poetry/lib/python3.10/site-packages/poetry/installation/installer.py:241 in _do_install
       239│                 source_root=self._env.path.joinpath("src")
       240│             ):
     → 241│                 ops = solver.solve(use_latest=self._whitelist).calculate_operations()
       242│         else:
       243│             self._io.write_line("Installing dependencies from lock file")

   7  ~/venvs/poetry/lib/python3.10/site-packages/poetry/puzzle/solver.py:71 in solve
        69│         with self._progress(), self._provider.use_latest_for(use_latest or []):
        70│             start = time.time()
     →  71│             packages, depths = self._solve()
        72│             end = time.time()
        73│ 

   6  ~/venvs/poetry/lib/python3.10/site-packages/poetry/puzzle/solver.py:154 in _solve
       152│ 
       153│         try:
     → 154│             result = resolve_version(self._package, self._provider)
       155│ 
       156│             packages = result.packages

   5  ~/venvs/poetry/lib/python3.10/site-packages/poetry/mixology/__init__.py:18 in resolve_version
        16│     solver = VersionSolver(root, provider)
        17│ 
     →  18│     return solver.solve()
        19│ 

   4  ~/venvs/poetry/lib/python3.10/site-packages/poetry/mixology/version_solver.py:175 in solve
       173│             while next is not None:
       174│                 self._propagate(next)
     → 175│                 next = self._choose_package_version()
       176│ 
       177│             return self._result()

   3  ~/venvs/poetry/lib/python3.10/site-packages/poetry/mixology/version_solver.py:532 in _choose_package_version
       530│ 
       531│         if not conflict:
     → 532│             self._solution.decide(package.package)
       533│             self._log(
       534│                 f"selecting {package.package.complete_name}"

   2  ~/venvs/poetry/lib/python3.10/site-packages/poetry/mixology/partial_solution.py:90 in decide
        88│         self._decisions[package.complete_name] = package
        89│ 
     →  90│         self._assign(
        91│             Assignment.decision(package, self.decision_level, len(self._assignments))
        92│         )

   1  ~/venvs/poetry/lib/python3.10/site-packages/poetry/mixology/partial_solution.py:115 in _assign
       113│         """
       114│         self._assignments.append(assignment)
     → 115│         self._register(assignment)
       116│ 
       117│     def backtrack(self, decision_level: int) -> None:

  AssertionError

  at ~/venvs/poetry/lib/python3.10/site-packages/poetry/mixology/partial_solution.py:151 in _register
      147│         name = assignment.dependency.complete_name
      148│         old_positive = self._positive.get(name)
      149│         if old_positive is not None:
      150│             value = old_positive.intersect(assignment)
    → 151│             assert value is not None
      152│             self._positive[name] = value
      153│ 
      154│             return
      155│

eaaltonen commented 6 months ago

Issue #5617 hits the same assertion, and could be related.

dimbleby commented 6 months ago

I do not know what this is trying to achieve but I am not at all surprised that it does not work

I would think that the best you can hope for from poetry here is a cleaner error.

MLNW commented 6 months ago

I'm hitting the same error. It worked without issue ~3 weeks ago. It also worked today until I tried to install a new dependency. Nothing else has changed in my environment.

MLNW commented 6 months ago

I found a work around here: https://github.com/vmware/vsphere-automation-sdk-python/issues/355#issuecomment-1870295459

DanCardin commented 5 months ago

I can reliably reproduce this with just docutils

[tool.poetry]
name = "example"
version = "0.0.0"
description = ""
authors = []

[tool.poetry.dependencies]
docutils = ">=0.19.0"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

If you poetry lock in an empty venv, I get

Package docutils (0.21.post1) not found.

If you then pip install docutils, then poetry lock, it produces:

  AssertionError

  at ~/.local/share/pypoetry/venv/lib/python3.12/site-packages/poetry/mixology/partial_solution.py:151 in _register
      147│         name = assignment.dependency.complete_name
      148│         old_positive = self._positive.get(name)
      149│         if old_positive is not None:
      150│             value = old_positive.intersect(assignment)
    → 151│             assert value is not None
      152│             self._positive[name] = value
      153│
      154│             return
      155│

I first ran into this locally, but have since tried it inside a vanilla python docker container, and it produced the same results.

simonrouse9461 commented 5 months ago

I’m trying to install apache-airflow with poetry. I ran into the same error because of the docutils dependency for airflow.

dimbleby commented 5 months ago

docutils is nothing much to do with this issue, which specifically covers the bizarro vsphere sdk stuff

the docutils thing will be caused by the mis-publishing of docutils 0.21.0 as per eg https://github.com/python-poetry/poetry/issues/9293, you'll want to exclude the bad docutils versions I expect

DanCardin commented 5 months ago

For whatever reason github search did not route me to #9293 (although that issue does seem to be more focused on actually addressing the former error that you get when docutils isnt yet in the venv); and i'm not particularly interested in the mis-published docutils issue itself.

This issue, other the other hand (from a layperson not familiar with poetry internals) seems more generally about dependency versions that hit this assertion. Regardless of the reason, the fact that poetry is bailing out of installation/locking with a bare assert and no information seems like the "bug" here, unless either case can be made to not fail at all.

Were the offending line instead:

# assert value is not None
if value is None:
    raise RuntimeError(f"Resolution failure: {old_positive} and {assignment} cannot be reconciled")

I probably would have fiddled with my version constraints until it worked rather than even searching github issues.

dimbleby commented 4 months ago

https://github.com/python-poetry/poetry/issues/9248#issuecomment-2026240520

bamiaux commented 4 months ago

A few notes:

it works on windows and install all dependencies. I’ve noticed that packages use a local url looking like file://c:/ something
It doesn’t work on Linux. I have looked at the code and just before the assert, it tries to compare two packages (class Package) for equality. If I tweak this specific check, it installs correctly on Linux too. The comparison fails because at some point it tries to compare two source_url but one is a local path looking like « /home/user/.cache/…/some.whl » and the other one is the same but with a random slash prefix added looking like « //home/user/.cache/…/some.whl »

I suspect if we remove this spurious ‘/‘ it will fix this bug

bamiaux commented 4 months ago

I did more debugging:

In vsphere setup.py we have the following:

setup(name='vsphere-automation-sdk',
      version='1.86.0',
      description='VMware vSphere Automation SDK for Python',
      url='https://github.com/vmware/vsphere-automation-sdk-python',
      author='VMware, Inc.',
      license='MIT',
      packages=[],
      install_requires=[
        'lxml >= 4.3.0',
        'pyVmomi >=6.7',
        'vapi-runtime @ file://localhost/{}/lib/vapi-runtime/vapi_runtime-2.44.0-py2.py3-none-any.whl'.format(os.getcwd()),
        'vcenter-bindings @ file://localhost/{}/lib/vcenter-bindings/vcenter_bindings-4.2.0-py2.py3-none-any.whl'.format(os.getcwd()),
        'vapi-common-client @ file://localhost/{}/lib/vapi-common-client/vapi_common_client-2.44.0-py2.py3-none-any.whl'.format(os.getcwd()),
        'vmwarecloud-aws @ file://localhost/{}/lib/vmwarecloud-aws/vmwarecloud_aws-1.64.0-py2.py3-none-any.whl'.format(os.getcwd()),
        'nsx-python-sdk @ file://localhost/{}/lib/nsx-python-sdk/nsx_python_sdk-4.1.2.0.0-py2.py3-none-any.whl'.format(os.getcwd()),
        'nsx-policy-python-sdk @ file://localhost/{}/lib/nsx-policy-python-sdk/nsx_policy_python_sdk-4.1.2.0.0-py2.py3-none-any.whl'.format(os.getcwd()),
        'nsx-vmc-policy-python-sdk @ file://localhost/{}/lib/nsx-vmc-policy-python-sdk/nsx_vmc_policy_python_sdk-4.1.2.0.0-py2.py3-none-any.whl'.format(os.getcwd()),
        'nsx-vmc-aws-integration-python-sdk @ file://localhost/{}/lib/nsx-vmc-aws-integration-python-sdk/nsx_vmc_aws_integration_python_sdk-4.1.2.0.0-py2.py3-none-any.whl'.format(os.getcwd()),
        'vmwarecloud-draas @ file://localhost/{}/lib/vmwarecloud-draas/vmwarecloud_draas-1.23.0-py2.py3-none-any.whl'.format(os.getcwd()),
      ]
)

So vmwarecloud-draas get setup as a requirement with an url looking like vmwarecloud-draas @ file://localhost//home/user/.../lib/vmwarecloud-draas/vmwarecloud_draas-1.23.0-py2.py3-none-any.whl

Note the duplicate '/' on localhost//home

Here's a commit which remove this duplicate '/' so install can move forward and complete successfully.

https://github.com/bamiaux/poetry/commit/ab5825c805cb00d3d04f011aa13ee8dc074d3463

I'm not particularly pleased with this fix, maybe there is a more elegant solution

python-poetry / poetry