poetry never resolves package combination: google-cloud-aiplatform kfp

Poetry version: 1.6.1
Python version: 3.11.5
OS version and name: docker image python:3.11.5
pyproject.toml: Providing reproducible docker command instead
[X] I am on the latest stable Poetry version, installed using a recommended method.
[X] I have searched the issues of this repo and believe that this is not a duplicate.
[X] I have consulted the FAQ and blog for any relevant entries or release notes.
[X] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

The following combination of packages never resolves: google-cloud-aiplatform kfp

Poetry is stuck in Resolving dependencies... after >10h.

Reproduced in both Native Macos 13.5.1 (22G90) and in Docker.

In Macos envs, cache and poetry.lock was removed like this, but despite this the error persists:

poetry env remove --all \
&& poetry cache clear --all -q . \
&& rm -rf $(poetry config cache-dir)/artifacts \
&& rm -f poetry.lock

Tested package combinations

❌ Doesn't resolve (within reasonable time)

google-cloud-aiplatform zenml mlflow gcsfs kfp <-- This combo was left running >10h
google-cloud-aiplatform kfp

✅ Does resolve

google-cloud-aiplatform zenml
google-cloud-aiplatform zenml mlflow
google-cloud-aiplatform zenml mlflow gcsfs
google-cloud-aiplatform gcsfs

How to reproduce

docker run --rm python:3.11.5 sh -c " \
curl -sSL https://install.python-poetry.org | python3 - --version=1.6.1 \
&& poetry --version \
&& mkdir app && cd app \
&& /root/.local/bin/poetry init --name=app --description=app --author=test@localhost --python='>=3.10,<3.12' -n \
&& /root/.local/bin/poetry add google-cloud-aiplatform kfp"

Output

Retrieving Poetry metadata

# Welcome to Poetry!

This will download and install the latest version of Poetry,
a dependency and package manager for Python.

It will add the `poetry` command to Poetry's bin directory, located at:

/root/.local/bin

You can uninstall at any time by executing this script with the --uninstall option,
and these changes will be reverted.

Installing Poetry (1.6.1)
Installing Poetry (1.6.1): Creating environment
Installing Poetry (1.6.1): Installing Poetry
Installing Poetry (1.6.1): Creating script
Installing Poetry (1.6.1): Done

Poetry (1.6.1) is installed now. Great!

To get started you need Poetry's bin directory (/root/.local/bin) in your `PATH`
environment variable.

Add `export PATH="/root/.local/bin:$PATH"` to your shell configuration file.

Alternatively, you can call Poetry explicitly with `/root/.local/bin/poetry`.

You can test that everything is set up by executing:

`poetry --version`

Creating virtualenv app-9TtSrW0h-py3.11 in /root/.cache/pypoetry/virtualenvs
Using version ^1.33.1 for google-cloud-aiplatform
Using version ^2.3.0 for kfp

Updating dependencies
Resolving dependencies...

Still present in 1.7.0

Now at Nov29, and poetry 1.7.1 the packages:

$poetry add google-cloud-pipeline-components kfp

It fails with:

The following error occurred when trying to handle this error:

  OverrideNeeded

  {Package('google-api-core', '2.14.0', features=frozenset({'grpc'})): {'grpcio': <Dependency grpcio (>=1.49.1,<2.0dev)>, 'grpcio-status': <Dependency grpcio-status (>=1.49.1,<2.0.dev0)>}, Package('google-cloud-bigquery', '3.13.0'): {'grpcio': <Dependency grpcio (>=1.49.1,<2.0dev)>}, Package('google-cloud-resource-manager', '1.10.4'): {'proto-plus': <Dependency proto-plus (>=1.22.2,<2.0.0dev)>}, Package('google-api-core', '2.13.1', features=frozenset({'grpc'})): {'grpcio': <Dependency grpcio (>=1.49.1,<2.0dev)>, 'grpcio-status': <Dependency grpcio-status (>=1.49.1,<2.0.dev0)>}, Package('google-api-core', '2.13.0', features=frozenset({'grpc'})): {'grpcio': <Dependency grpcio (>=1.49.1,<2.0dev)>, 'grpcio-status': <Dependency grpcio-status (>=1.49.1,<2.0.dev0)>}, Package('google-api-core', '2.12.0', features=frozenset({'grpc'})): {'grpcio': <Dependency grpcio (>=1.49.1,<2.0dev)>, 'grpcio-status': <Dependency grpcio-status (>=1.49.1,<2.0.dev0)>}, Package('google-api-core', '2.11.1', features=frozenset({'grpc'})): {'grpcio': <Dependency grpcio (>=1.49.1,<2.0dev)>, 'grpcio-status': <Dependency grpcio-status (>=1.49.1,<2.0.dev0)>}, Package('google-api-core', '2.11.0', features=frozenset({'grpc'})): {'grpcio': <Dependency grpcio (>=1.49.1,<2.0dev)>, 'grpcio-status': <Dependency grpcio-status (>=1.49.1,<2.0dev)>}, Package('google-cloud-resource-manager', '1.10.3'): {'proto-plus': <Dependency proto-plus (>=1.22.2,<2.0.0dev)>}, Package('google-cloud-resource-manager', '1.10.2'): {'proto-plus': <Dependency proto-plus (>=1.22.2,<2.0.0dev)>}, Package('google-cloud-resource-manager', '1.10.1'): {'proto-plus': <Dependency proto-plus (>=1.22.2,<2.0.0dev)>}, Package('google-cloud-resource-manager', '1.10.0'): {'proto-plus': <Dependency proto-plus (>=1.22.2,<2.0.0dev)>}, Package('google-cloud-resource-manager', '1.9.1'): {'proto-plus': <Dependency proto-plus (>=1.22.2,<2.0.0dev)>}, Package('google-cloud-resource-manager', '1.9.0'): {'proto-plus': <Dependency proto-plus (>=1.22.2,<2.0.0dev)>}, Package('google-cloud-resource-manager', '1.8.1'): {'proto-plus': <Dependency proto-plus (>=1.22.2,<2.0.0dev)>}, Package('google-cloud-resource-manager', '1.8.0'): {'proto-plus': <Dependency proto-plus (>=1.22.2,<2.0.0dev)>}}

  at ~/.local/share/pypoetry/venv/lib/python3.11/site-packages/poetry/puzzle/provider.py:660 in complete_package
      656│                     current_overrides.update({dependency_package: package_overrides})
      657│                     overrides.append(current_overrides)
      658│ 
      659│             if overrides:
    → 660│                 raise OverrideNeeded(*overrides)
      661│ 
      662│         # Modifying dependencies as needed
      663│         clean_dependencies = []
      664│         for dep in dependencies:

The following error occurred when trying to handle this error:

  AssertionError

Intersting that the following (change from python 3.11 to 3.10) works without any problems:

[tool.poetry]
name = "NA"
version = "0.1.0"
description = "NA"
authors = ["David Cardozo <david.cardozo@me.com>"]
readme = "README.md"
packages = [{include = "merge_vertex"}]

[tool.poetry.dependencies]
python = "~3.10"
kfp = "^2.4.0"

[tool.poetry.group.dev.dependencies]
ipykernel = "^6.20.2"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

I have the same problem:

Poetry version: 1.7.1
Python version: 3.10.12
OS version and name: Ubuntu 22.04.3 LTS

pyproject.toml


[tool.poetry]
name = "my-project"
version = "0.0.1"
description = ""
authors = []
readme = "README.md"
packages = [{ include = "my_pkg" }]

[tool.poetry.dependencies] python = ">=3.10,<3.12" kfp = "1.8.22" google-cloud-aiplatform = "1.25.0"

[build-system] requires = ["poetry-core"] build-backend = "poetry.core.masonry.api"


## Issue

To reproduce the issue using the same environment, just run `poetry install`

It will fall into infinite dependency resolving. 
```bash
$ poetry install
Creating virtualenv my-project in /home/.../.venv
Updating dependencies
Resolving dependencies... (2003.3s)

The logs using -vvv option shows what is the package that is causing the problem:

Found 0 compatible packages for grpcio-status
   1: fact: no versions of grpcio-status match >1.49.1,<1.50.0 || >1.50.0,<1.51.0rc1 || >1.51.0rc1,<1.51.1 || >1.51.1,<1.51.3 || >1.51.3,<1.52.0rc1 || >1.52.0rc1,<1.53.0 || >1.53.0,<1.53.1 || >1.53.1,<1.53.2 || >1.53.2,<1.54.0 || >1.54.0,<1.54.2 || >1.54.2,<1.54.3 || >1.54.3,<1.55.0rc1 || >1.55.0rc1,<1.55.3 || >1.55.3,<1.56.0 || >1.56.0,<1.56.2 || >1.56.2,<1.57.0 || >1.57.0,<1.58.0 || >1.58.0,<1.59.0 || >1.59.0,<1.59.2 || >1.59.2,<1.59.3 || >1.59.3,<1.60.0 || >1.60.0,<1.60.1 || >1.60.1,<2.0.dev0
   1: conflict: no versions of grpcio-status match >1.49.1,<1.50.0 || >1.50.0,<1.51.0rc1 || >1.51.0rc1,<1.51.1 || >1.51.1,<1.51.3 || >1.51.3,<1.52.0rc1 || >1.52.0rc1,<1.53.0 || >1.53.0,<1.53.1 || >1.53.1,<1.53.2 || >1.53.2,<1.54.0 || >1.54.0,<1.54.2 || >1.54.2,<1.54.3 || >1.54.3,<1.55.0rc1 || >1.55.0rc1,<1.55.3 || >1.55.3,<1.56.0 || >1.56.0,<1.56.2 || >1.56.2,<1.57.0 || >1.57.0,<1.58.0 || >1.58.0,<1.59.0 || >1.59.0,<1.59.2 || >1.59.2,<1.59.3 || >1.59.3,<1.60.0 || >1.60.0,<1.60.1 || >1.60.1,<2.0.dev0
   1: ! grpcio-status (>1.49.1,!=1.50.0,!=1.51.0rc1,!=1.51.1,!=1.51.3,!=1.52.0rc1,!=1.53.0,!=1.53.1,!=1.53.2,!=1.54.0,!=1.54.2,!=1.54.3,!=1.55.0rc1,!=1.55.3,!=1.56.0,!=1.56.2,!=1.57.0,!=1.58.0,!=1.59.0,!=1.59.2,!=1.59.3,!=1.60.0,!=1.60.1,<2.0.dev0) is partially satisfied by not grpcio-status (==1.51.0rc1)
   1: ! which is caused by "grpcio-status (1.51.0rc1) depends on protobuf (>=4.21.6)"
   1: ! thus: grpcio-status (>1.49.1,<1.50.0 || >1.50.0,<1.51.1 || >1.51.1,<1.51.3 || >1.51.3,<1.52.0rc1 || >1.52.0rc1,<1.53.0 || >1.53.0,<1.53.1 || >1.53.1,<1.53.2 || >1.53.2,<1.54.0 || >1.54.0,<1.54.2 || >1.54.2,<1.54.3 || >1.54.3,<1.55.0rc1 || >1.55.0rc1,<1.55.3 || >1.55.3,<1.56.0 || >1.56.0,<1.56.2 || >1.56.2,<1.57.0 || >1.57.0,<1.58.0 || >1.58.0,<1.59.0 || >1.59.0,<1.59.2 || >1.59.2,<1.59.3 || >1.59.3,<1.60.0 || >1.60.0,<1.60.1 || >1.60.1,<2.0.dev0) requires protobuf (>=4.21.6)
   1: ! grpcio-status (>1.49.1,!=1.50.0,!=1.51.1,!=1.51.3,!=1.52.0rc1,!=1.53.0,!=1.53.1,!=1.53.2,!=1.54.0,!=1.54.2,!=1.54.3,!=1.55.0rc1,!=1.55.3,!=1.56.0,!=1.56.2,!=1.57.0,!=1.58.0,!=1.59.0,!=1.59.2,!=1.59.3,!=1.60.0,!=1.60.1,<2.0.dev0) is partially satisfied by not grpcio-status (==1.52.0rc1)
   1: ! which is caused by "grpcio-status (1.52.0rc1) depends on protobuf (>=4.21.6)"
...
   1: conflict: grpcio-status (1.56.2) depends on protobuf (>=4.21.6)
...
   1: conflict: grpcio-status (1.60.0) depends on protobuf (>=4.21.6)
...

Tons of those conflits appears, but no error or exception is raised.

The `pip` way

But, when I use built-in package venv instead of poetry it runs smootly.

$ python3 -m venv .venv
$ source .venv/bin/activate
(.venv) $ pip3 install kfp==1.8.22 google-cloud-aiplatform==1.25.0

And the most important, pip can found a version for grpcio-status...

(.venv) $ pip3 show grpcio-status
Name: grpcio-status
Version: 1.48.2
Summary: Status proto mapping for gRPC
Home-page: https://grpc.io
Author: The gRPC Authors
Author-email: grpc-io@googlegroups.com
License: Apache License 2.0
Location: /home/.../.venv/lib/python3.10/site-packages
Requires: googleapis-common-protos, grpcio, protobuf
Required-by:

Attempts

I've tried to use the following attempts with no sucess:

Force grpcio-status = "1.48.2" in pyproject.toml
Be flexible with dependencies using kfp = "^1.8" and google-cloud-aiplatform = "^1"
Clear all caches by running poetry cache clear pypi --all and rm -rf ~/.cache/pypoetry/cache

Sum up

In short terms: this dependencies in pyproject.toml can't be installed using poetry, but they can using pip

Extra information

This dependencies could be installed normally in macOS Sonoma 14.1.1
The updated versions of kfp v2.x and google-cloud-platform it seems to work fine. The issue occurs in kfp v1.x only.

Intersting that the following (change from python 3.11 to 3.10) works without any problems:

[tool.poetry]
name = "NA"
version = "0.1.0"
description = "NA"
authors = ["David Cardozo <david.cardozo@me.com>"]
readme = "README.md"
packages = [{include = "merge_vertex"}]

[tool.poetry.dependencies]
python = "~3.10"
kfp = "^2.4.0"

[tool.poetry.group.dev.dependencies]
ipykernel = "^6.20.2"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

@Davidnet, could you try to install both kfp and google-cloud-aiplatform now? Here it works, kfp released v2.7.0 recently, just want to be sure that issue is kfp v1.x related.

Still present in Poetry 1.8.3

I have had the same problem for a long time. My suggestion for the moment is to install both with pip and wait for any returns on this issue.

python-poetry / poetry