search

pypa / pipenv

Python Development Workflow for Humans.
https://pipenv.pypa.io
MIT License
24.88k stars 1.87k forks source link

Pipenv installs spurious dependencies #4371

Closed J-M0 closed 2 years ago

J-M0 commented 4 years ago

Issue description

When installing any version of Django < 3.0 as a dependency and django-debug-toolbar as a dev dependency, pipenv lock determines that asgiref is a required package when it in fact is not.

Expected result

$ pipenv graph
django-debug-toolbar==2.2
  - Django [required: >=1.11, installed: 2.2.13]
    - pytz [required: Any, installed: 2020.1]
    - sqlparse [required: Any, installed: 0.3.1]
  - sqlparse [required: >=0.2.0, installed: 0.3.1]

Actual result

$ pipenv graph
asgiref==3.2.10
django-debug-toolbar==2.2
  - Django [required: >=1.11, installed: 2.2.13]
    - pytz [required: Any, installed: 2020.1]
    - sqlparse [required: Any, installed: 0.3.1]
  - sqlparse [required: >=0.2.0, installed: 0.3.1]

Steps to replicate

In a new project, run the following commands:

$ pipenv install 'Django<3.0'
$ pipenv install --dev django-debug-toolbar
$ pipenv --support Pipenv version: `'2020.6.2'` Pipenv location: `'/usr/local/Cellar/pipenv/2020.6.2/libexec/lib/python3.8/site-packages/pipenv'` Python location: `'/usr/local/Cellar/pipenv/2020.6.2/libexec/bin/python3.8'` Python installations found: - `3.8.0`: `/Users/jmorris/.pyenv/versions/3.8.0/bin/python3` - `3.7.7`: `/usr/local/bin/python3` - `3.7.7`: `/usr/local/bin/python3.7m` - `3.7.7`: `/usr/local/bin/python3.7` - `3.6.8`: `/Users/jmorris/.pyenv/versions/3.6.8/bin/python3` - `3.5.9`: `/Users/jmorris/.pyenv/versions/3.5.9/bin/python3` - `3.5.1`: `/Users/jmorris/.pyenv/versions/3.5.1/bin/python3` - `2.7.18`: `/Users/jmorris/.pyenv/versions/2.7.18/bin/python2` - `2.7.17`: `/usr/local/bin/python2` - `2.7.17`: `/usr/local/bin/python2.7` - `2.7.16`: `/usr/bin/python2.7` PEP 508 Information: ``` {'implementation_name': 'cpython', 'implementation_version': '3.8.3', 'os_name': 'posix', 'platform_machine': 'x86_64', 'platform_python_implementation': 'CPython', 'platform_release': '18.7.0', 'platform_system': 'Darwin', 'platform_version': 'Darwin Kernel Version 18.7.0: Mon Apr 27 20:09:39 PDT ' '2020; root:xnu-4903.278.35~1/RELEASE_X86_64', 'python_full_version': '3.8.3', 'python_version': '3.8', 'sys_platform': 'darwin'} ``` System environment variables: - `PATH` - `TERM_PROGRAM` - `TERM` - `SHELL` - `TMPDIR` - `Apple_PubSub_Socket_Render` - `TERM_PROGRAM_VERSION` - `TERM_SESSION_ID` - `USER` - `SSH_AUTH_SOCK` - `__CF_USER_TEXT_ENCODING` - `PWD` - `EDITOR` - `LANG` - `ITERM_PROFILE` - `XPC_FLAGS` - `XPC_SERVICE_NAME` - `SHLVL` - `PYENV_SHELL` - `HOME` - `COLORFGBG` - `LC_TERMINAL_VERSION` - `ITERM_SESSION_ID` - `LOGNAME` - `LESS` - `LESSOPEN` - `LC_TERMINAL` - `COLORTERM` - `PIP_DISABLE_PIP_VERSION_CHECK` - `PYTHONDONTWRITEBYTECODE` - `PIP_SHIMS_BASE_MODULE` - `PIP_PYTHON_PATH` - `PYTHONFINDER_IGNORE_UNSUPPORTED` Pipenv–specific environment variables: Debug–specific environment variables: - `PATH`: `/usr/local/Cellar/pipenv/2020.6.2/libexec/tools:/Users/jmorris/.pyenv/shims:/Users/jmorris/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion.app/Contents/Public:/usr/local/MacGPG2/bin:/Applications/Wireshark.app/Contents/MacOS` - `SHELL`: `/usr/local/bin/fish` - `EDITOR`: `vim` - `LANG`: `en_US.UTF-8` - `PWD`: `/tmp/lol` --------------------------- Contents of `Pipfile` ('/private/tmp/lol/Pipfile'): ```toml [[source]] name = "pypi" url = "https://pypi.org/simple" verify_ssl = true [dev-packages] django-debug-toolbar = "*" [packages] django = "<3.0" [requires] python_version = "3.8" ``` Contents of `Pipfile.lock` ('/private/tmp/lol/Pipfile.lock'): ```json { "_meta": { "hash": { "sha256": "5e37dff5e53a743305538fdd5fb9da0dbe7eea0521b520e50a56ee191050c830" }, "pipfile-spec": 6, "requires": { "python_version": "3.8" }, "sources": [ { "name": "pypi", "url": "https://pypi.org/simple", "verify_ssl": true } ] }, "default": { "django": { "hashes": [ "sha256:84f370f6acedbe1f3c41e1a02de44ac206efda3355e427139ecb785b5f596d80", "sha256:e8fe3c2b2212dce6126becab7a693157f1a441a07b62ec994c046c76af5bb66d" ], "index": "pypi", "version": "==2.2.13" }, "pytz": { "hashes": [ "sha256:a494d53b6d39c3c6e44c3bec237336e14305e4f29bbf800b599253057fbb79ed", "sha256:c35965d010ce31b23eeb663ed3cc8c906275d6be1a34393a1d73a41febf4a048" ], "version": "==2020.1" }, "sqlparse": { "hashes": [ "sha256:022fb9c87b524d1f7862b3037e541f68597a730a8843245c349fc93e1643dc4e", "sha256:e162203737712307dfe78860cc56c8da8a852ab2ee33750e33aeadf38d12c548" ], "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'", "version": "==0.3.1" } }, "develop": { "asgiref": { "hashes": [ "sha256:7e51911ee147dd685c3c8b805c0ad0cb58d360987b56953878f8c06d2d1c6f1a", "sha256:9fc6fb5d39b8af147ba40765234fa822b39818b12cc80b35ad9b0cef3a476aed" ], "markers": "python_version >= '3.5'", "version": "==3.2.10" }, "django": { "hashes": [ "sha256:84f370f6acedbe1f3c41e1a02de44ac206efda3355e427139ecb785b5f596d80", "sha256:e8fe3c2b2212dce6126becab7a693157f1a441a07b62ec994c046c76af5bb66d" ], "index": "pypi", "version": "==2.2.13" }, "django-debug-toolbar": { "hashes": [ "sha256:eabbefe89881bbe4ca7c980ff102e3c35c8e8ad6eb725041f538988f2f39a943", "sha256:ff94725e7aae74b133d0599b9bf89bd4eb8f5d2c964106e61d11750228c8774c" ], "index": "pypi", "version": "==2.2" }, "pytz": { "hashes": [ "sha256:a494d53b6d39c3c6e44c3bec237336e14305e4f29bbf800b599253057fbb79ed", "sha256:c35965d010ce31b23eeb663ed3cc8c906275d6be1a34393a1d73a41febf4a048" ], "version": "==2020.1" }, "sqlparse": { "hashes": [ "sha256:022fb9c87b524d1f7862b3037e541f68597a730a8843245c349fc93e1643dc4e", "sha256:e162203737712307dfe78860cc56c8da8a852ab2ee33750e33aeadf38d12c548" ], "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3'", "version": "==0.3.1" } } } ```
stianjensen commented 4 years ago

Not sure why it ends up at the top-level, but django does require asgiref in setup.cfg: https://github.com/django/django/blob/197b55c53469cf8344d1ba35175236780cb83bd1/setup.cfg#L43-L46

J-M0 commented 4 years ago

Yes, Django started requiring asgiref in Django 3. If you look at the requirements for Django 2.2.x, asgiref is absent:

https://github.com/django/django/blob/stable/2.2.x/setup.py#L86

GPHemsley-RELX commented 2 years ago

The linked #4455 provides more context: The packages that are being installed are future dependencies; that is, dependencies of later versions of packages that --keep-outdated is, well, keeping outdated. They shouldn't even enter into the dependency tree if their parent package is being held back.

This is still a problem in pipenv 2022.1.8.

matteius commented 2 years ago

@GPHemsley-RELX Can you check if it is still a problem in my branch that upgrades to latest pip? Branch is vendor-pip-22.0.3-followup-changes and the new pip resolver solves some other infinity lock issues so I am wondering if the resolver fixes that as well.

EDIT: Nevermind, I just checked the steps it appears to still be a problem even on my branch:

matteius@matteius-VirtualBox:~/shared-projects/pipenv-triage/pipenv-4371$ pipenv graph
asgiref==3.5.0
django-debug-toolbar==3.2.4
  - Django [required: >=2.2, installed: 2.2.27]
    - pytz [required: Any, installed: 2021.3]
    - sqlparse [required: >=0.2.2, installed: 0.4.2]
  - sqlparse [required: >=0.2.0, installed: 0.4.2]

Just noting this is specifically an issue of installing a new dependency as --dev because if I repeat the steps but install pipenv install django-debug-toolbar regularly it produces the expected graph.

dqkqd commented 2 years ago

@matteius My current solution is creating default constraints associated with packages, then pass those constraints to resolver and use them when installing dev-packages. But it only solves the case when installing from Pipfile, not the case where user provides packages.

# Pipfile
[packages]
django = "<3.0"

[dev-packages]
django-debug-toolbar = "*"
$ pipenv graph
django-debug-toolbar==3.2.4
  - Django [required: >=2.2, installed: 2.2.28]
    - pytz [required: Any, installed: 2022.1]
    - sqlparse [required: >=0.2.2, installed: 0.4.2]
  - sqlparse [required: >=0.2.0, installed: 0.4.2]

Should I open a PR just for this case?

matteius commented 2 years ago

New version of pipenv==2022.8.13 has been released with the fix produced by @dqkqd !