Tons of new dependencies introduced in requirements for delphi-epidata (python client)

[boxydog]

Thanks for your work on epidata.

Between delphi-epidata 4.1.20 and 4.1.23, you seem to have introduced at least 66 new dependencies, see below. (I have a few in our project already, so you may have introduced more than 66.)

I am concerned that introducing all these new dependencies into my production application will make it less likely to be reliable.

Is there any way to get a more slimmed down version of delphi-epidata?

$ diff tmp/[ab]
5a6
> astroid
6a8,9
> boto3
> botocore
7a11
> cffi
8a13,22
> clarabel
> click
> click-plugins
> cligj
> contourpy
> coverage[toml]
> covidcast
> cryptography
> cvxpy
> cycler
9a24,25
> delphi-utils
> descartes
16a33
> ecos
17a35,37
> fiona
> fonttools
> freezegun
18a39,41
> geopandas
> gitdb
> gitpython
23a47,52
> imageio
> imageio-ffmpeg
> iniconfig
> isort
> jinja2
> jmespath
25a55,61
> kiwisolver
> lazy-object-proxy
> markupsafe
> matplotlib
> mccabe
> mock
> moto
28a65
> osqp
29a67,69
> pandas
> pillow
> pluggy
30a71
> pycparser
32a74,83
> pydocstyle
> pylint
> pyparsing
> pyproj
> pytest
> pytest-cov
> python-dateutil
> pytz
> pyyaml
> qdldl
34a86,88
> requests-mock
> responses
> s3transfer
36a91,95
> scs
> shapely
> six
> slackclient
> smmap
37a97
> snowballstemmer
38a99
> structlog
41a103
> toml
44a107
> werkzeug
45a109,111
> wrapt
> xlrd
> xmltodict

[boxydog]

It might be from more closely tying delphi-utils to this repo, perhaps #1460?

[dshemetov]

Thank you for the report. This will take us a bit of time to address. A few possible workarounds for now:

• pin the delphi-epidata client version to 4.1.20; the main update in 4.1.23 is some logging needed for our internal use; that client version should stay compatible for the foreseeable future (we can let you know in this thread if that changes)
• you can also copy this main client file to your project directory and directly import it; you will then need to comment out this line and this line

Let me know if the first works for you for now and meanwhile we can see what we can do to slim the dependencies on our end.

[boxydog]

Thanks so much for your quick reply. I'm happy to pin to 4.1.20 for now, sounds like that could work for awhile.

[melange396]

@boxydog you are exactly right, it is from adding the dependency of delphi-utils in #1460. The change may add some to the installation footprint, but it should not affect its reliability much if at all; the only functionality used from the added library is in logging.

[boxydog]

@boxydog you are exactly right, it is from adding the dependency of delphi-utils in #1460. The change may add some to the installation footprint, but it should not affect its reliability much if at all; the only functionality used from the added library is in logging.

Thanks for your reply.

If I include 66 more libraries, then any one of them can have a bug, incompatible version, or other problem that makes it harder just to install the delphi-epidata library. That's what I mean by affecting reliability, not that the code path will have a bug, but that the dependency will cause support burden. In general, I don't want to rely on code I don't use.

So I hope it's possible to look into slimming down the dependencies to only what delphi-epidata needs. I understand it can be non-trivial to do so.

[dshemetov]

@melange396 I started thinking how we could address this and got a couple ideas:

• we could bundle logger.py from this repo with the Python client (maybe by copying in the packaging step)
• an idea I'm more partial to is unbundling our logger.py from delphi-utils into a separate delphi_logger package+repo, put it on PyPI and install from there; this would solve the annoyance of needing to keep these files synchronized

[melange396]

I dont know if its worth it to create a new PyPI package for a dinky little logging config... I am very into deduplicating the logger.py file though -- the acquisition and server code can use delphi_utils.logger instead of delphi.epidata.common.logger, especially because they both already use delphi_utils in some way or another.

Logging in the client is a fringe case for debugging usage at this point, and as a user facing package, it doesnt really gain much from rigid log-like formatting. What if we just dumb down the logging in the client for now? (I created PR https://github.com/cmu-delphi/delphi-epidata/pull/1486 to demonstrate.) Then, at some point after we take the bloat out of delphi_utils, we might once again consider using it for logging in the client.