search

pypa / pipenv

Python Development Workflow for Humans.
https://pipenv.pypa.io
MIT License
24.88k stars 1.87k forks source link

Unexpected Dependancy Graph Issue #2529

Closed JBKahn closed 6 years ago

JBKahn commented 6 years ago
Issue description

I had a requirement pinned to "<4.0.0" and a sub-dependancy that required it to be "==3.3.0". I checked the graph and as expected, that was the only dependancy that required the protobuf package.

That gave me a dependancy resolution error of: Could not find a version that matches protobuf<4.0.0,==3.3.0,==3.6.0

Expected result

I expected it to resolve to "==3.3.0"

Actual result

I was able to get it to be happy by changing my local "<4.0.0" to "==3.3.0".

Steps to replicate

I will fill this is later today, but wanted to spell out the issue just in case it is known.

Provide the steps to replicate (which usually at least includes the commands and the Pipfile).

Please run $ pipenv --support, and paste the results here. Don't put backticks (`) around it! The output already contains Markdown formatting.

If you're on macOS, run the following:

$ pipenv --support | pbcopy

If you're on Windows, run the following:

> pipenv --support | clip
erinxocon commented 6 years ago

Hey @JBKahn Any update on this?

JBKahn commented 6 years ago

I had been slammed at work, it seems to be possibly related to my build environment as this didn't happen on my machine but inside an alpine python docker container. Was going to dig in before I included all the details.

teeberg commented 6 years ago

I think I'm experiencing the same issue with a different package:

Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies. You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation. Hint: try $ pipenv lock --pre if it is a pre-release dependency. Could not find a version that matches botocore<1.11.0,==1.10.26,>=1.10.26,>=1.10.64 (from -r /var/folders/h4/n2jm49b97dx5d89gkkt4yxkm0000gn/T/pipenv-KF21xr-requirements/pipenv-Vk6UhE-constraints.txt (line 34))

and it's true, ==1.10.26 and >=1.10.64 contradict themselves, but no package seems to be pulling in >=1.10.64?!

After running the suggested pipenv install --skip-lock and inspecting the output of pipenv graph, here are botocore's dependency specs:

> pipenv graph | grep -i botocore
  - botocore [required: ==1.10.26, installed: 1.10.26]
    - botocore [required: >=1.3.0,<2.0.0, installed: 1.10.26]
  - botocore [required: >=1.0.1, installed: 1.10.26]
    - botocore [required: >=1.10.26,<1.11.0, installed: 1.10.26]
      - botocore [required: >=1.3.0,<2.0.0, installed: 1.10.26]

I'm just investigating how much work it is to switch over to using pipenv from a workflow using pip-compile, so I have another virtual env set up with exactly these requirements as specified, and using pipdeptree in there doesn't show anything specifying >=1.10.64 either:

> pipdeptree -r -p botocore | grep "requires: botocore"
  - awscli==1.15.26 [requires: botocore==1.10.26]
  - awsebcli==3.14.3 [requires: botocore>=1.0.1]
  - boto3==1.7.26 [requires: botocore>=1.10.26,<1.11.0]
  - s3transfer==0.1.12 [requires: botocore>=1.3.0,<2.0.0]

So, maybe this comes down to, how do I find out where that bogus version spec comes from?

JBKahn commented 6 years ago

yeah, the real difficulty is that when there is a collision, the output isn't showing where the different requirements come from. I've been super busy these past few days but I'll try to get to this ASAP.

mr-bo-jangles commented 6 years ago

yeah, the real difficulty is that when there is a collision, the output isn't showing where the different requirements come from

Literally dealing with this right now, it's impossible to track down which package is causing issues

mr-bo-jangles commented 6 years ago
$ pipenv --support Pipenv version: `'2018.7.1'` Pipenv location: `'/home/josh/.virtualenvs/savings-champion/lib/python3.6/site-packages/pipenv'` Python location: `'/home/josh/.virtualenvs/savings-champion/bin/python'` Other Python installations in `PATH`: - `2.7`: `/usr/bin/python2.7` - `2.7`: `/usr/bin/python2.7` - `3.6`: `/usr/bin/python3.6m` - `3.6`: `/home/josh/.virtualenvs/savings-champion/bin/python3.6` - `3.6`: `/usr/bin/python3.6` - `3.6.6`: `/home/josh/.virtualenvs/savings-champion/bin/python` - `3.6.6`: `/usr/bin/python` - `2.7.15`: `/usr/bin/python2` - `3.6.6`: `/home/josh/.virtualenvs/savings-champion/bin/python3` - `3.6.6`: `/usr/bin/python3` PEP 508 Information: ``` {'implementation_name': 'cpython', 'implementation_version': '3.6.6', 'os_name': 'posix', 'platform_machine': 'x86_64', 'platform_python_implementation': 'CPython', 'platform_release': '4.17.11-arch1', 'platform_system': 'Linux', 'platform_version': '#1 SMP PREEMPT Sun Jul 29 10:11:16 UTC 2018', 'python_full_version': '3.6.6', 'python_version': '3.6', 'sys_platform': 'linux'} ``` System environment variables: - `PATH` - `XAUTHORITY` - `GDMSESSION` - `XDG_DATA_DIRS` - `ZDOTDIR` - `KDE_SESSION_UID` - `TERM` - `DBUS_SESSION_BUS_ADDRESS` - `LANG` - `XDG_SESSION_TYPE` - `XDG_SESSION_ID` - `XDG_CURRENT_DESKTOP` - `QT_AUTO_SCREEN_SCALE_FACTOR` - `DISPLAY` - `MAIL` - `MOZ_PLUGIN_PATH` - `SESSION_MANAGER` - `USERNAME` - `LOGNAME` - `PWD` - `_` - `XCURSOR_THEME` - `LANGUAGE` - `KDE_SESSION_VERSION` - `SHELL` - `GDM_LANG` - `XCURSOR_SIZE` - `DESKTOP_SESSION` - `OLDPWD` - `USER` - `DESKTOP_STARTUP_ID` - `KDE_FULL_SESSION` - `WINDOWPATH` - `XDG_SEAT` - `GTK_MODULES` - `GS_LIB` - `LC_ALL` - `KWIN_TRIPLE_BUFFER` - `XDG_RUNTIME_DIR` - `XDG_SESSION_DESKTOP` - `XDG_VTNR` - `SHLVL` - `HOME` - `EDITOR` - `VISUAL` - `PAGER` - `LESS` - `LESS_TERMCAP_mb` - `LESS_TERMCAP_md` - `LESS_TERMCAP_me` - `LESS_TERMCAP_se` - `LESS_TERMCAP_so` - `LESS_TERMCAP_ue` - `LESS_TERMCAP_us` - `LS_COLORS` - `GREP_COLOR` - `GREP_COLORS` - `VIRTUAL_ENV` - `PS1` - `PYTHONDONTWRITEBYTECODE` - `PIP_PYTHON_PATH` Pipenv–specific environment variables: Debug–specific environment variables: - `PATH`: `/home/josh/.virtualenvs/savings-champion/bin:/usr/local/bin:/usr/local/sbin:/home/josh/go/bin:/home/josh/.yarn/bin:/usr/bin:/opt/cuda/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl` - `SHELL`: `/usr/bin/zsh` - `EDITOR`: `nano` - `LANG`: `en_GB.UTF-8` - `PWD`: `/home/josh/PycharmProjects/savings-champion` - `VIRTUAL_ENV`: `/home/josh/.virtualenvs/savings-champion` --------------------------- Contents of `Pipfile` ('/home/josh/PycharmProjects/savings-champion/Pipfile'): ```toml [[source]] url = "https://pypi.org/simple" verify_ssl = true name = "pypi" [packages] lxml = "*" nose = "*" django-extensions = "*" django-tinymce = "*" sorl-thumbnail = "*" "suds-py3" = "*" feedparser = "*" python-dateutil = "*" python-memcached = "*" django-pipeline = "*" requests = "*" django-pipeline-forgiving = "*" django-ckeditor = "*" django-filebrowser = "*" raven = "*" celery = {version = "*", extras = ["redis"]} "psycopg2-binary" = "*" django-crispy-forms = "*" pyotp = "*" newrelic = "*" djangorestframework = "*" djangorestframework-xml = "*" djangorestframework-jwt = "*" django-filter = "*" parsedatetime = "*" simple-salesforce = "*" django-allauth = "*" apache-libcloud = "*" django-storages = "*" "boto3" = "*" sortedcontainers = "*" redis = "*" hiredis = "*" msgpack-python = "*" django-waffle = "*" django-report-builder = "*" django-markupfield = "*" django-cors-headers = "*" arrow = "*" django-redis = "*" "mailchimp3" = "*" dotmailer = {editable = true, git = "https://github.com/savingschampion/dotmailer.git"} simplejson = "*" zeep = "*" djangorestframework-jsonp = "*" requests-oauthlib = "*" django-ipware = "*" hypothesis = {version = "*", extras = ["pytz"]} drf-yasg = {version = "*", extras = ["validation"]} Django = ">=2.0" Pillow = "*" uWSGI = "*" "Django-Select2" = "*" Collectfast = "*" Markdown = "*" Faker = "*" rest_condition = "*" django-rest-knox = "*" django-simple-history = "*" wagtail = "*" wagtail-metadata-mixin = "*" [dev-packages] [requires] python_version = "3.6" ``` 
Locking [dev-packages] dependencies...
Locking [packages] dependencies...

Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
Could not find a version that matches Django<2.1,==2.1,>=1.11,>=1.11.7,>=1.3,>=1.8,>=2.0 (from -r /tmp/pipenv-0s0uam0c-requirements/pipenv-7uweinlt-constraints.txt (line 51))
Tried: 1.1.3, 1.1.4, 1.2, 1.2.1, 1.2.2, 1.2.3, 1.2.4, 1.2.5, 1.2.6, 1.2.7, 1.3, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5, 1.3.6, 1.3.7, 1.4, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 1.4.5, 1.4.6, 1.4.7, 1.4.8, 1.4.9, 1.4.10, 1.4.11, 1.4.12, 1.4.13, 1.4.14, 1.4.15, 1.4.16, 1.4.17, 1.4.18, 1.4.19, 1.4.20, 1.4.21, 1.4.22, 1.5, 1.5.1, 1.5.2, 1.5.2, 1.5.3, 1.5.4, 1.5.5, 1.5.6, 1.5.7, 1.5.8, 1.5.8, 1.5.9, 1.5.10, 1.5.11, 1.5.12, 1.5.12, 1.6, 1.6, 1.6.1, 1.6.1, 1.6.2, 1.6.2, 1.6.3, 1.6.3, 1.6.4, 1.6.4, 1.6.5, 1.6.5, 1.6.6, 1.6.6, 1.6.7, 1.6.7, 1.6.8, 1.6.8, 1.6.9, 1.6.9, 1.6.10, 1.6.10, 1.6.11, 1.6.11, 1.7, 1.7, 1.7.1, 1.7.1, 1.7.2, 1.7.2, 1.7.3, 1.7.3, 1.7.4, 1.7.4, 1.7.5, 1.7.5, 1.7.6, 1.7.6, 1.7.7, 1.7.7, 1.7.8, 1.7.8, 1.7.9, 1.7.9, 1.7.10, 1.7.10, 1.7.11, 1.7.11, 1.8, 1.8, 1.8.1, 1.8.1, 1.8.2, 1.8.2, 1.8.3, 1.8.3, 1.8.4, 1.8.4, 1.8.5, 1.8.5, 1.8.6, 1.8.6, 1.8.7, 1.8.7, 1.8.8, 1.8.8, 1.8.9, 1.8.9, 1.8.10, 1.8.10, 1.8.11, 1.8.11, 1.8.12, 1.8.12, 1.8.13, 1.8.13, 1.8.14, 1.8.14, 1.8.15, 1.8.15, 1.8.16, 1.8.16, 1.8.17, 1.8.17, 1.8.18, 1.8.18, 1.8.19, 1.8.19, 1.9, 1.9, 1.9.1, 1.9.1, 1.9.2, 1.9.2, 1.9.3, 1.9.3, 1.9.4, 1.9.4, 1.9.5, 1.9.5, 1.9.6, 1.9.6, 1.9.7, 1.9.7, 1.9.8, 1.9.8, 1.9.9, 1.9.9, 1.9.10, 1.9.10, 1.9.11, 1.9.11, 1.9.12, 1.9.12, 1.9.13, 1.9.13, 1.10, 1.10, 1.10.1, 1.10.1, 1.10.2, 1.10.2, 1.10.3, 1.10.3, 1.10.4, 1.10.4, 1.10.5, 1.10.5, 1.10.6, 1.10.6, 1.10.7, 1.10.7, 1.10.8, 1.10.8, 1.11, 1.11, 1.11.1, 1.11.1, 1.11.2, 1.11.2, 1.11.3, 1.11.3, 1.11.4, 1.11.4, 1.11.5, 1.11.5, 1.11.6, 1.11.6, 1.11.7, 1.11.7, 1.11.8, 1.11.8, 1.11.9, 1.11.9, 1.11.10, 1.11.10, 1.11.11, 1.11.11, 1.11.12, 1.11.12, 1.11.13, 1.11.13, 1.11.14, 1.11.14, 1.11.15, 1.11.15, 2.0, 2.0, 2.0.1, 2.0.1, 2.0.2, 2.0.2, 2.0.3, 2.0.3, 2.0.4, 2.0.4, 2.0.5, 2.0.5, 2.0.6, 2.0.6, 2.0.7, 2.0.7, 2.0.8, 2.0.8, 2.1, 2.1
Skipped pre-versions: 1.8a1, 1.8b1, 1.8b2, 1.8rc1, 1.9a1, 1.9b1, 1.9rc1, 1.9rc2, 1.10a1, 1.10a1, 1.10b1, 1.10b1, 1.10rc1, 1.10rc1, 1.11a1, 1.11b1, 1.11rc1, 1.11rc1, 2.0a1, 2.0b1, 2.0rc1, 2.1a1, 2.1b1, 2.1rc1
There are incompatible versions in the resolved dependencies.

I have no idea where the ==2.1 comes from, listing of all requirements from graph with Django in the name

  - Django [required: >=1.8, installed: 2.0.7]
  - django-storages [required: >=1.6, installed: 1.6.6]
django-allauth==0.36.0
  - Django [required: >=1.11, installed: 2.0.7]
django-ckeditor==5.6.1
  - django-js-asset [required: Any, installed: 1.1.0]
django-cors-headers==2.4.0
django-crispy-forms==1.7.2
django-extensions==2.1.0
django-filebrowser==3.10.1
  - django-grappelli [required: >=2.11,<2.12, installed: 2.11.1]
django-filter==2.0.0
  - Django [required: >=1.11, installed: 2.0.7]
django-ipware==2.1.0
django-markupfield==1.4.3
django-pipeline-forgiving==1.0.0
  - django-pipeline [required: Any, installed: 1.6.14]
django-redis==4.9.0
  - Django [required: >=1.11, installed: 2.0.7]
django-report-builder==6.0.2
  - djangorestframework [required: >=3.1.0, installed: 3.8.2]
django-rest-knox==3.1.5
  - django [required: Any, installed: 2.0.7]
  - djangorestframework [required: Any, installed: 3.8.2]
django-select2==6.1.1
  - django-appconf [required: >=0.6.0, installed: 1.0.2]
django-simple-history==2.3.0
django-tinymce==2.7.0
django-url-filter==0.3.5
  - Django [required: >=1.8, installed: 2.0.7]
django-waffle==0.14.0
djangorestframework-jsonp==1.0.2
djangorestframework-jwt==1.11.0
djangorestframework-xml==1.3.0
  - Django [required: >=1.11.7, installed: 2.0.7]
  - djangorestframework [required: >=3.7.7, installed: 3.8.2]
  - django [required: >=1.3, installed: 2.0.7]
  - djangorestframework [required: Any, installed: 3.8.2]
  - django-meta [required: >=1.0, installed: 1.4.1]
    - Django [required: >=1.11,<2.1, installed: 2.0.7]
    - django-modelcluster [required: >=4.0,<5.0, installed: 4.1]
    - django-taggit [required: >=0.22.2,<1.0, installed: 0.22.2]
    - django-treebeard [required: >=4.2.0,<5.0, installed: 4.3]
      - Django [required: >=1.8, installed: 2.0.7]
    - djangorestframework [required: >=3.7.4,<4.0, installed: 3.8.2]
uranusjr commented 6 years ago

It’s from Wagtail, you need to pin django<2.1. (I know this because we are working on this right now and this can be pulled out from the half-down code.)

And yes, I know the resolver should be able to pin this automatically, but right now it’s what it is.

JBKahn commented 6 years ago

Is it easy to write tests for the resolver. I haven't taken a look at the code but I'd help out if it isn't too bad to get started.

OliverHofkens commented 6 years ago

I'm running into the same issue with boto3:

$ pipenv lock
Could not find a version that matches botocore<1.11.0,<1.12.0,>=1.10.84,>=1.11.3

Running pipenv install --skip-lock and inspecting the graph shows that no package seems to require <1.11.0

$ pipenv graph | grep botocore
    - botocore [required: >=1.11.3,<1.12.0, installed: 1.11.3]
      - botocore [required: >=1.3.0,<2.0.0, installed: 1.11.3]
        - botocore [required: >=1.11.3,<1.12.0, installed: 1.11.3]
          - botocore [required: >=1.3.0,<2.0.0, installed: 1.11.3]
mjdunn commented 6 years ago

@OliverHofkens see #2596. Running pipenv lock --clear helped me.

OliverHofkens commented 6 years ago

@mjdunn Great, thanks!

teeberg commented 6 years ago

still seeing this, but with a different package now:

Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  First try clearing your dependency cache with $ pipenv lock --clear, then try the original command again.
 Alternatively, you can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
  Hint: try $ pipenv lock --pre if it is a pre-release dependency.
Could not find a version that matches html5lib==1.0b10,>=1.0
Tried: 1.0-reupload, 0.2, 0.9, 0.10, 0.11, 0.11.1, 0.90, 0.95, 0.99, 0.999, 0.9999, 0.99999, 0.999999, 0.9999999, 0.99999999, 0.999999999, 0.999999999, 1.0.1, 1.0.1
Skipped pre-versions: 1.0b1, 1.0b2, 1.0b3, 1.0b5, 1.0b6, 1.0b7, 1.0b8, 1.0b9, 1.0b10, 1.0b10
There are incompatible versions in the resolved dependencies.

but according to pipdeptree | grep -i html5lib, nothing pulls in >=1.0:

    - html5lib [required: ==1.0b10, installed: 1.0b10]
  - html5lib [required: Any, installed: 1.0b10]
techalchemy commented 6 years ago

With which version of pipenv

teeberg commented 6 years ago

sorry, missed that! in 2018.10.9. I think that's the latest, right?