Loosen dependency requirements for dxpy package

katiesandford commented 1 year ago

By having strict dependency requirements, it is hard to install dxpy as a library as it is likely to clash with other python libraries. AFAIK it is best practice as a library builder to make your dependencies as loose as possible (as long as the code still works of course).

Could you please loosen the requirement on the following libraries:

requests - 2.28.0 came out in June 2022 (although the latest is 2.28.2) and is required by some other python libraries.
websocket-client - too strict a requirement
colorama and pyreadline3. Although these are only a requirement for windows, the dependency resolver poetry will ensure dependencies are resolved for all platforms, and so this is limiting for poetry users on mac or linux.

I don't personally care about the dependencies pinned for python < 3.10 but others might, so it might be good to loosen those too.

Current requirements.txt:

argcomplete>=2.0.0; python_version >= "3.10"
argcomplete>=1.9.4,<2.0.0; python_version < "3.10"
websocket-client==0.54.0
python-dateutil>=2.5
psutil>=5.9.3
requests>=2.8.0,<=2.27.1
cryptography==3.3.2; python_version < "3"
cryptography==3.2.1; python_version > "3" and python_version < "3.6"
cryptography>=3.4.2,<37; python_version > "3.5"
gnureadline==8.0.0; sys_platform == "darwin" and python_version < "3.9" and platform_machine == "x86_64"
pyreadline==2.1; sys_platform == "win32" and python_version < "3.5"
pyreadline3==3.4.1; sys_platform == "win32" and python_version >= "3.5"
colorama==0.4.4; sys_platform == "win32"

daquang commented 1 year ago

Would also like to see a loosening of requirements, especially for requests

TriLe965 commented 1 year ago

Thanks for bringing this up. We are looking into this issue.

@katiesandford can you help us to elaborate more on the items below?

colorama and pyreadline3. Although these are only a requirement for windows, the dependency resolver poetry will ensure dependencies are resolved for all platforms, and so this is limiting for poetry users on mac or linux."

and

websocket-client==0.54.0

FYI, our engineers followed instructions on Poetry's website for Ubuntu 22.04. We were able to add dx-toolkit as a dependency without any complaints about unresolvable dependencies.

$ curl -sSL https://install.python-poetry.org | python3 -
$ export PATH="/root/.local/bin:$PATH"
$ poetry --version
Poetry (version 1.4.1)
$ poetry new poetry-demo
Created package poetry_demo in poetry-demo
$ cd poetry-demo/
$ poetry add dxpy
Creating virtualenv poetry-demo-HvxGlNIk-py3.10 in /root/.cache/pypoetry/virtualenvs
Using version ^0.343.0 for dxpy

Updating dependencies
Resolving dependencies... (25.7s)

Writing lock file

Package operations: 15 installs, 0 updates, 0 removals

  • Installing pycparser (2.21)
  • Installing certifi (2022.12.7)
  • Installing cffi (1.15.1)
  • Installing charset-normalizer (2.0.12)
  • Installing idna (3.4)
  • Installing six (1.16.0)
  • Installing urllib3 (1.26.15)
  • Installing argcomplete (3.0.5)
  • Installing backports-ssl-match-hostname (3.5.0.1)
  • Installing cryptography (36.0.2)
  • Installing psutil (5.9.4)
  • Installing python-dateutil (2.8.2)
  • Installing requests (2.27.1)
  • Installing websocket-client (0.54.0)
  • Installing dxpy (0.343.0)
$ poetry install
Installing dependencies from lock file

No dependencies to install or update

Installing the current project: poetry-demo (0.1.0)
$ poetry show
argcomplete                  3.0.5     Bash tab completion for argparse
backports-ssl-match-hostname 3.5.0.1   The ssl.match_hostname() function from Python 3.5
certifi                      2022.12.7 Python package for providing Mozilla's CA Bundle.
cffi                         1.15.1    Foreign Function Interface for Python calling C code.
charset-normalizer           2.0.12    The Real First Universal Charset Detector. Open, modern and actively maintained alternative to Chardet.
cryptography                 36.0.2    cryptography is a package which provides cryptographic recipes and primitives to Python developers.
dxpy                         0.343.0   DNAnexus Platform API bindings for Python
idna                         3.4       Internationalized Domain Names in Applications (IDNA)
psutil                       5.9.4     Cross-platform lib for process and system monitoring in Python.
pycparser                    2.21      C parser in Python
python-dateutil              2.8.2     Extensions to the standard Python datetime module
requests                     2.27.1    Python HTTP for Humans.
six                          1.16.0    Python 2 and 3 compatibility utilities
urllib3                      1.26.15   HTTP library with thread-safe connection pooling, file post, and more.
websocket-client             0.54.0    WebSocket client for Python. hybi13 is supported.
$ poetry build
Building poetry-demo (0.1.0)
  - Building sdist
  - Built poetry_demo-0.1.0.tar.gz
  - Building wheel
  - Built poetry_demo-0.1.0-py3-none-any.whl

It would be really helpful if you can show us a way to reproduce your difficulty.

emiloslavsky commented 1 year ago

@katiesandford ping

katiesandford commented 1 year ago

Hi. The problem arises when trying to install dxpy alongside other python packages. Here is one example where I've tried to install it with a later version of requests, but the problem is general. e.g if any other package I want to install relies on a recent version of requests or on websocket-client that doesn't exactly match the version you require, dxpy cannot be installed.

poetry --version
Poetry (version 1.3.2)
poetry new poetry-demo
Created package poetry_demo in poetry-demo
cd poetry-demo
poetry add requests>2.28.0
poetry add dxpy
Using version ^0.344.0 for dxpy

Updating dependencies
Resolving dependencies... Downloading https://files.pythonhosted.org/packages/f0/77/6b602e0aa5baa928cb6d4462cc2325514c7c5008f8ddd
Resolving dependencies... (0.4s)

Because no versions of dxpy match >0.344.0,<0.345.0
 and dxpy (0.344.0) depends on requests (>=2.8.0,<=2.27.1), dxpy (>=0.344.0,<0.345.0) requires requests (>=2.8.0,<=2.27.1).
So, because poetry-demo depends on both requests (^2.28.2) and dxpy (^0.344.0), version solving failed.

wormsik commented 1 year ago

Hi @katiesandford, we have recently bumped all dependencies to the most recent versions and we plan to cut a new dx-toolkit release soon. Hope this will solve your issue with requests library.

There is still one exception as websocket-client still contains one major unresolved bug but we plan to update as soon as it will be finally resolved.

Regarding system-specific dependencies - unfortunately, we are not able to remove them as dx-toolkit is used on various platforms and we need to ensure we can provide the same user experience on all of them. In case you would experience any issues with dependency resolution in Poetry (which seems to handle platform markers incorrectly), you can add dxpy with --platform <linux/darwin> (https://python-poetry.org/docs/cli/#add) argument which would restrict dxpy only for your desired environment:

root@ubuntu:/# poetry new test
Created package test in test
root@ubuntu:/# cd test/
root@ubuntu:/test# poetry add dxpy --platform linux
Using version ^0.345.0 for dxpy

Updating dependencies
Resolving dependencies... (1.5s)

Writing lock file

...
root@ubuntu:/test# poetry add colorama=="0.4.6"  # It is successfull even though this colorama version is not compatible with dxpy 0.345.0

Updating dependencies
Resolving dependencies... (0.3s)

Writing lock file

...

DCGenomics commented 1 year ago

@hillwithsmallfields @eckersleyj

dnanexus / dx-toolkit

Loosen dependency requirements for dxpy package #941