search

python-poetry / poetry

Python packaging and dependency management made easy
https://python-poetry.org
MIT License
31.71k stars 2.27k forks source link

Fix dependency versions does not prevent duplicate dependency resolution process. #6089

Closed krlng closed 1 year ago

krlng commented 2 years ago

Infos:

Description: The pyproject I linked is a minimal version of our real pyorject.yaml (see below). It contains two major dependencies, that both rely on different versions of numpy depending on the python-version, platform, and chipset you are on:

pandas:

install_requires =
    numpy>=1.18.5; platform_machine!='aarch64' and platform_machine!='arm64' and python_version<'3.10'
    numpy>=1.19.2; platform_machine=='aarch64' and python_version<'3.10'
    numpy>=1.20.0; platform_machine=='arm64' and python_version<'3.10'
    numpy>=1.21.0; python_version>='3.10'

opencv-python:

  "numpy==1.13.3; python_version=='3.6' and platform_machine != 'aarch64' and platform_machine != 'arm64'",
  "numpy==1.14.5; python_version=='3.7' and platform_machine != 'aarch64' and platform_machine != 'arm64'",
  "numpy==1.17.3; python_version=='3.8' and platform_machine != 'aarch64' and platform_machine != 'arm64'",
  "numpy==1.19.3; python_version<='3.9' and sys_platform == 'linux' and platform_machine == 'aarch64'",
  "numpy==1.21.0; python_version<='3.9' and sys_platform == 'darwin' and platform_machine == 'arm64'",
  "numpy==1.19.3; python_version=='3.9' and platform_machine != 'aarch64' and platform_machine != 'arm64'",
  "numpy==1.21.2; python_version>='3.10'"

When you resolve the project (poetry lock -vvv), poetry realizes that there are 16 different possible combinations and resolves all of them.

Excerpt of the logs:


   0: Duplicate dependencies for numpy
   0: Different requirements found for numpy (>=1.21.2) with markers python_version >= "3.6" and platform_system == "Darwin" and platform_machine == "arm64", numpy (>=1.19.3) with markers python_version >= "3.6" and platform_system == "Linux" and platform_machine == "aarch64" or python_version >= "3.9", numpy (>=1.14.5) with markers python_version >= "3.7" and numpy (>=1.17.3) with markers python_version >= "3.8".
   1: Version solving took 0.902 seconds.
   1: Tried 1 solutions.
   0: Retrying dependency resolution with the following overrides ({Package('opencv-python-headless', '4.6.0.66'): {'numpy': <Dependency numpy (>=1.21.2)>}}).
   1: fact: resolve-dependencies is 0.1.0
   1: derived: resolve-dependencies
   1: fact: resolve-dependencies depends on opencv-python-headless (^4.6)
   1: fact: resolve-dependencies depends on pandas (^1.3)
   1: fact: resolve-dependencies depends on numpy (=1.21.2)
   1: selecting resolve-dependencies (0.1.0)
   1: derived: numpy (==1.21.2)
   1: derived: pandas (>=1.3,<2.0)
   1: derived: opencv-python-headless (>=4.6,<5.0)
   1: selecting numpy (1.21.2)
   1: fact: opencv-python-headless (4.6.0.66) depends on numpy (>=1.21.2)
   1: selecting opencv-python-headless (4.6.0.66)
PyPI: Getting info for pandas (1.4.3) from PyPI
   0: Duplicate dependencies for numpy
   0: Different requirements found for numpy (>=1.18.5) with markers platform_machine != "aarch64" and platform_machine != "arm64" and python_version < "3.10", numpy (>=1.19.2) with markers platform_machine == "aarch64" and python_version < "3.10" and numpy (>=1.20.0) with markers platform_machine == "arm64" and python_version < "3.10".
   1: Version solving took 0.138 seconds.
   1: Tried 1 solutions.

This will continue for 15 more strategies, so in the end, there will be these combinations:


({
    Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.21 .2) >  }
}), ({
    Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.21 .2) >  },
    Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.18 .5) > }
}), ({
    Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.21 .2) >  },
    Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.19 .2) > }
}), ({
    Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.21 .2) >  },
    Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.20 .0) > }
}), ({
    Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.19 .3) >  }
}), ({
    Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.19 .3) >  },
    Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.18 .5) > }
}), ({
    Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.19 .3) >  },
    Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.19 .2) > }
}), ({
    Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.19 .3) >  },
    Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.20 .0) > }
}), ({
    Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.14 .5) >  }
}), ({
    Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.14 .5) >  },
    Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.18 .5) > }
}), ({
    Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.14 .5) >  },
    Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.19 .2) > }
}), ({
    Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.14 .5) >  },
    Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.20 .0) > }
}), ({
    Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.17 .3) >  }
}), ({
    Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.17 .3) >  },
    Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.18 .5) > }
}), ({
    Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.17 .3) >  },
    Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.19 .2) > }
}), ({
    Package('opencv-python-headless', '4.6.0.66'): {'numpy': < Dependency numpy( >= 1.17 .3) >  },
    Package('pandas', '1.4.3'): {'numpy': < Dependency numpy( >= 1.20 .0) > }
})

It can quickly be seen, that no library has a more upper bound than numpy < 2.0, so using the newest version of numpy should just be fine. As the resolution system does not realize that on its own, I pinned my numpy to 1.21.2. However, this simply gets ignored, and the process is exactly the same, no matter if I specified this.

Why is this the case? I would suspect it is related to the different platform_machine and sys_platform markers. This is why I also tried to pin numpy for every possible combination of those, but this also did not help.

Is there any way to prevent poetry from resolving all those different strategies?

While this is more a small blemish in the posted example, it is a major issue in our real use case, as that one includes multiple packages living in a private package repository, which will interrupt the connection after a few minutes because of hitting the rate limit, which makes it practically impossible to use poetry at all.

krlng commented 2 years ago

Any comments or ideas? Would not like to see this issue going stale

radoering commented 2 years ago

It can quickly be seen, that no library has a more upper bound than numpy < 2.0, so using the newest version of numpy should just be fine.

That's too short-sighted. Even if there is no upper bound, it is possible that a (transient) dependency of the latest numpy version conflicts with another dependency for a specific platform marker so that an older numpy version has to be used for that platform.

I pinned my numpy to 1.21.2. However, this simply gets ignored, and the process is exactly the same, no matter if I specified this.

That's just another constraint. It probably helps to avoid backtracking, so that the only version that is tried for all overrides is numpy 1.21.2 and if there is a conflict, no other versions are tried.

Is there any way to prevent poetry from resolving all those different strategies?

At the moment you can only restrict the python version (which you have already done in the given pyproject.toml). This avoids those overrides where the python version does not match. (I'm not sure how good this works in poetry 1.1.x so you may have to try 1.2 beta.)

If only specific platforms are relevant to you, you may be interested in #4956.

github-actions[bot] commented 8 months ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.