Poetry will lock with two different versions of pandas if numpy is pinned

[J-France]

• [x] I am on the latest Poetry version.

• [x] I have searched the issues of this repo and believe that this is not a duplicate.

• [ ] ~If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).~

• OS version and name: Ubuntu 20.04

• Poetry version: 1.1.11

• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/J-France/70df79b776c2e5722eb37773472d5d92

Issue

Poetry will lock to two different versions of pandas if numpy is set to a fixed version. The following command will install pandas 1.3.4 into the virtual environment

poetry add numpy=1.20.1 pandas="*"

However the lock file contains two versions (1.3.3 and 1.3.4) as well as various poetry commands returning different versions

• poetry export returns 1.3.3
• poetry show return 1.3.4
• poetry show --tree returns both 1.3.3 and 1.3.4

[zachliu]

not sure if it's related, i just had this error msg:

  SolverProblemError

  Because pandas (1.3.4) depends on numpy (>=1.21.0)
   and <my-stuff> depends on numpy (1.20.3), pandas is forbidden.
  So, because <my-stuff> depends on pandas (1.3.4), version solving failed.

then i checked pandas 1.3.4, it doesn't actually depends on numpy >= 1.21.0 if i'm using python 3.8 https://github.com/pandas-dev/pandas/blob/945c9ed766a61c7d2c0a7cbb251b6edebf9cb7d5/setup.cfg#L35-L38 it seems poetry is unable to recognize the new syntax there, so only the last row numpy >= 1.21.0 is registered :thinking:

[setu4993]

it seems poetry is unable to recognize the new syntax there, so only the last row numpy >= 1.21.0 is registered 🤔.

I think @zachliu is onto something here.

I'm seeing SolverProblemError when trying to install awswrangler in a project with python = "^3.6.2,<3.10". awswrangler specifies different versions of pandas for 3.6 and 3.7 respectively as:

pandas = [
    { version = "~1.1.0", optional = true, python = "~=3.6.2" },
    { version = "^1.2.0", optional = true, python = ">=3.7.1,<3.10" },
]

Which can also be confirmed by looking at the metadata in the distributed package by running curl -sL https://pypi.org/pypi/awswrangler/json | jq '.info.requires_dist':

[
...
  "pandas (>=1.1.0,<1.2.0); python_full_version ~= \"3.6.2\"",
  "pandas (>=1.2.0,<2.0.0); python_full_version >= \"3.7.1\" and python_full_version < \"4.0.0\"",
...
]

[setu4993]

This is still a problem, any help?

[zachliu]

same here... i noticed this issue because i've been trying to transform apache/airflow provided constraints file into a pyproject.toml

They have

pandas==1.3.4
numpy==1.20.3

in the constraint file. When I added them into my pyproject.toml I just got this SolverProblemError error. Right now I have to keep pandas==1.3.3

[dimbleby]

pandas later than 1.3.3 have numpy>=1.21.0; python_version>='3.10'. This is not compatible with the requested numpy=1.20.1 (nor, in the most recent comment, with numpy==1.20.3)

That is: numpy==1.20.3 and pandas==1.3.4 are not compatible if you're supporting python >= 3.10.

re the original report, the solver seems to get into some sort of muddle where it goes:

• pandas 1.4.2 looks good
• let's try with the override numpy >= 1.21.0
• that solves, but uses pandas 1.3.3
• er, so I guess we have both versions of pandas

I'm not sure how this is supposed to work, I suspect that the solution poetry ought to find would be to use pandas 1.3.3.

[artingithug]

Tip in of some info from my encountering the same issue:

• Environmental constraint:

  $ poetry --version
  Poetry (version 1.5.1)
  
  $ python --version
  Python 3.10.11
  
  $ grep -E '^python' pyproject.toml 
  python = "^3.10"

• Failure of pandas addition:

  $ poetry add pandas
  Using version ^2.1.3 for pandas
  
  Updating dependencies
  
  Resolving dependencies... (2.6s)
  Because no versions of pandas match >2.1.3,<3.0.0
   and pandas (2.1.3) depends on numpy (>=1.26.0,<2), pandas (>=2.1.3,<3.0.0) requires numpy (>=1.26.0,<2).
  And because checkthechain (0.3.9) depends on numpy (>=1.19.0,<1.24)
   and no versions of checkthechain match >0.3.9,<0.4.0, pandas (>=2.1.3,<3.0.0) is incompatible with checkthechain (>=0.3.9,<0.4.0).
  So, because dex-spot-price-aggregator depends on both checkthechain (^0.3.9) and pandas (^2.1.3), version solving failed.

• Gathered info of version of numpy dependency for each pandas' version:

  • Dependencies of pandas==2.1.3: ['numpy<2,>=1.22.4; python_version < "3.11"', 'numpy<2,>=1.23.2; python_version == "3.11"', 'numpy<2,>=1.26.0; python_version >= "3.12"', ...
  • Dependencies of pandas==2.1.2: ['numpy<2,>=1.22.4; python_version < "3.11"', 'numpy<2,>=1.23.2; python_version == "3.11"', 'numpy<2,>=1.26.0; python_version >= "3.12"' ...
  • Dependencies of pandas==2.1.1: ['numpy>=1.22.4; python_version < "3.11"', 'numpy>=1.23.2; python_version == "3.11"', 'numpy>=1.26.0; python_version >= "3.12"' ...
  • Dependencies of pandas==2.1.0: ['numpy>=1.22.4; python_version < "3.11"', 'numpy>=1.23.2; python_version >= "3.11"' ...
  • Dependencies of pandas==2.1.0rc0: ['numpy>=1.22.4; python_version < "3.11"', 'numpy>=1.23.2; python_version >= "3.11"', ...
  • Dependencies of pandas==2.0.3: ['python-dateutil (>=2.8.2)', 'pytz (>=2020.1)', 'tzdata (>=2022.1)', 'numpy (>=1.20.3) ; python_version < "3.10"', 'numpy (>=1.21.0) ; python_version >= "3.10"', 'numpy (>=1.23.2) ; python_version >= "3.11"' ...

• Unexpectedly excess pandas version 2.1.3 is also installed at attempt of adding pandas@^2.1.0 as workaround of the 'numpy<2,>=1.26.0; python_version >= "3.12"' dependency condition of pandas.

  $ poetry add pandas@^2.1.0
  
  Updating dependencies
  Resolving dependencies... (3.9s)
  
  Package operations: 3 installs, 0 updates, 0 removals
  
    • Installing python-dateutil (2.8.2)
    • Installing pytz (2023.3.post1)
    • Installing pandas (2.1.3)
  
  Writing lock file
  
  $ grep -H -n 'pandas' pyproject.toml 
  pyproject.toml:17:pandas = "^2.1.0"
  
  $ poetry show --tree
  ...
  pandas 2.1.0 Powerful data structures for data analysis, time series, and statistics
  ├── numpy >=1.23.2
  ├── python-dateutil >=2.8.2
  │   └── six >=1.5 
  ├── pytz >=2020.1
  └── tzdata >=2022.1
  pandas 2.1.3 Powerful data structures for data analysis, time series, and statistics
  ├── numpy >=1.22.4,<2
  ├── numpy >=1.23.2,<2
  ├── python-dateutil >=2.8.2
  │   └── six >=1.5 
  ├── pytz >=2020.1
  └── tzdata >=2022.1
  
  $ grep -A1 -E '^name = "numpy"' poetry.lock
  name = "numpy"
  version = "1.23.5"

  Excerpts about pandas from poetry.lock:

  [[package]]
  name = "pandas"
  version = "2.1.0"
  description = "Powerful data structures for data analysis, time series, and statistics"
  optional = false
  python-versions = ">=3.9"
  files = [
      {file = "pandas-2.1.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:40dd20439ff94f1b2ed55b393ecee9cb6f3b08104c2c40b0cb7186a2f0046242"},
      ...
      {file = "pandas-2.1.0.tar.gz", hash = "sha256:62c24c7fc59e42b775ce0679cfa7b14a5f9bfb7643cfbe708c960699e05fb918"},
  ]
  
  [package.dependencies]
  numpy = {version = ">=1.23.2", markers = "python_version >= \"3.11\""}
  ...
  
  [[package]]
  name = "pandas"
  version = "2.1.3"
  description = "Powerful data structures for data analysis, time series, and statistics"
  optional = false
  python-versions = ">=3.9"
  files = [
      {file = "pandas-2.1.3-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:acf08a73b5022b479c1be155d4988b72f3020f308f7a87c527702c5f8966d34f"},
      ...
      {file = "pandas-2.1.3.tar.gz", hash = "sha256:22929f84bca106921917eb73c1521317ddd0a4c71b395bcf767a106e3494209f"},
  ]
  
  [package.dependencies]
  numpy = [
      {version = ">=1.22.4,<2", markers = "python_version < \"3.11\""},
      {version = ">=1.23.2,<2", markers = "python_version == \"3.11\""},
  ]
  ...

My temporary workaround was obviously changing python version constrain in pyproject.toml.

[tool.poetry.dependencies]
# Due to poetry's issue 4703 (https://github.com/python-poetry/poetry/issues/4703)
# modified python version constrain to install pandas.
#python = "^3.10"
python = ">=3.10,<3.12"

$ poetry add pandas                                                                                                                                                                            
Using version ^2.1.3 for pandas                                                                                                                                                                                    

Updating dependencies                                                                                                                                                                                              
Resolving dependencies... (1.7s)                                                                                                                                                                                   

Package operations: 3 installs, 0 updates, 0 removals                                                                                                                                                              

  • Installing python-dateutil (2.8.2)                                                                                                                                                                             
  • Installing pytz (2023.3.post1)                                                                       
  • Installing pandas (2.1.3)                                                                                                                                                                                      

Writing lock file 

$ poetry show --tree | grep -A3 -E '^pandas'
pandas 2.1.3 Powerful data structures for data analysis, time series, and statistics
├── numpy >=1.22.4,<2
├── numpy >=1.23.2,<2
├── python-dateutil >=2.8.2