urllib3 dependency issue

[mhham]

The installation of the last dvc version (0.40.2) with google storage (pip install --no-cache-dir dvc[gs]) randomly fails because of a dependency conflict for urllib3 between requests ('urllib3>=1.21.1,<1.25') and dulwich ('urllib3[secure]>=1.23').

More precisely when installing dulwich first, it installs the last urllib3 version (1.25.2) which conflicts with requests.

For reproducibility, here is the Dockerfile that I use:

FROM python:3.7-alpine

RUN apk add --no-cache zlib-dev jpeg-dev gcc musl-dev \
 && pip install -U pip \
 && pip install --no-cache-dir dvc[gs]

related to https://github.com/pypa/pip/issues/988

Short term solution: Add urllib3==1.24.3 to setup.py and requirements

[mhham]

I would suggest checking the indirect dependencies of the package using tools like pip-compile before pushing a new release.

Or maybe rely on more complete python package management tools like Pipenv or Poetry

Example output of pip-compileon the current master branch, where we see that the workingurllib3 version is 1.24.3 and NOT the last 1.25.2:

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile
#

appdirs==1.4.3
asciimatics==1.11.0
certifi==2019.3.9         # via dulwich, requests
chardet==3.0.4            # via requests
colorama==0.4.1
configobj==5.0.6
configparser==3.7.4
contextlib2==0.5.5        # via schema
decorator==4.4.0          # via jsonpath-ng, networkx
distro==1.4.0
dulwich==0.19.11
future==0.17.1
gitdb2==2.0.5             # via gitpython
gitpython==2.1.11
grandalf==0.6
humanize==0.5.1
idna==2.8                 # via requests
inflect==2.1.0
jsonpath-ng==1.4.3
nanotime==0.5.2
networkx==2.3
pillow==6.0.0             # via asciimatics
ply==3.11
pyasn1==0.4.5
pyfiglet==0.8.post1       # via asciimatics
pyparsing==2.4.0          # via grandalf
requests==2.21.0
ruamel.yaml==0.15.94
schema==0.7.0
six==1.12.0               # via configobj, jsonpath-ng
smmap2==2.0.5             # via gitdb2
treelib==1.5.5
urllib3==1.24.3           # via dulwich, requests
wcwidth==0.1.7            # via asciimatics
zc.lockfile==1.4

# The following packages are considered to be unsafe in a requirements file:
# setuptools

[efiop]

@mhham Thanks for reporting this issue! We'll take a look.

[ghost]

Hopefully, this will be solved when they merge this patch: https://github.com/kennethreitz/requests/pull/5092

[mhham]

Hopefully, this will be solved when they merge this patch: kennethreitz/requests#5092

Indeed!

It still might be useful to adopt a more systematic approach for dependency resolution for the future (cf. https://github.com/iterative/dvc/issues/2008#issuecomment-492657143)

[ghost]

@mhham , thanks for such a detail description and suggestions on how to solve it! I created an issue to integrate a tool for it in our CI, so we don't publish anything with conflicting deps: https://github.com/iterative/dvc/issues/2014

We will deploy a new version today that requires requests to be >= 2.22.0, just to force the dependency resolution.

By the way, if you are interested in providing a patch for #2014 we are happy to help with what we can :)