Managing `poetry.lock` and `requirements.txt`

[kshalot]

@alistairewj @bemoody @tompollard Like we saw, there seems to be an inconsistency in the way we manage the dependencies (requirements.txt in particular).

The way we approached this in Health Data Nexus is:

1. Add the dependency to pyproject.toml
2. Generate a new poetry.lock via for example poetry lock --no-update
3. Generate a new requirements.txt via poetry export -f requirements.txt --output requirements.txt --with dev

Step 3 seems to embellish the file with extra markers (see https://github.com/MIT-LCP/physionet-build/pull/1867#discussion_r1106019930). This is more of a discussion starter than an issue report. We should probably decide on a specific process to get this done to now get surprising results.

[bemoody]

$ diff -u <(git show 8b5980b2260339a4eebb4d23f7c1937d063e0184:./requirements.txt | grep == | sed 's| \\||' | tr A-Z a-z | sort) <(git show ad4757d35d1e8e4180aca31c200d6334fcce1c20:./requirements.txt | grep == | sed 's| ; python_version >= "3.8" and python_version < "4[.0]*" \\$||' | sed 's| ; python_version < "4[.0]*" and python_version >= "3.8" \\||' | tr A-Z a-z | sort)
--- /dev/fd/63  2023-02-14 12:22:03.488557908 -0500
+++ /dev/fd/62  2023-02-14 12:22:03.488557908 -0500
@@ -1,52 +1,56 @@
 asgiref==3.5.2
-asn1crypto==0.24.0
+backports-zoneinfo==0.2.1 ; python_version >= "3.8" and python_version < "3.9" \
 bleach==3.3.0
-cachetools==3.1.1
+cachetools==4.2.2
 certifi==2022.12.7
 cffi==1.15.1
 chardet==3.0.4
 charset-normalizer==2.0.12
-coverage==4.4.2
+coverage==4.5.4
 cryptography==36.0.1
 django-autocomplete-light==3.9.4
 django-background-tasks-updated==1.2.7
 django-ckeditor==6.5.1
-django-compat==1.0.15
 django-debug-toolbar==3.2.4
 django-js-asset==2.0.0
 django-sass==1.1.0
-django-storages==1.12.3
+django-storages[google]==1.12.3
 django==4.1.6
 djangorestframework==3.14.0
-google-api-core==1.30.0
-google-api-python-client==1.7.9
+google-api-core==1.34.0
+google-api-core[grpc]==1.34.0
+google-api-python-client==1.12.8
 google-auth-httplib2==0.1.0
-google-auth==1.25.0
+google-auth==1.32.0
 google-cloud-core==1.7.0
 google-cloud-storage==1.42.3
+google-cloud-workflows==1.9.1
 google-crc32c==1.1.2
 google-resumable-media==1.3.1
-googleapis-common-protos==1.6.0
+googleapis-common-protos==1.58.0
+grpcio-status==1.48.2
+grpcio==1.51.1
+hdn-research-environment==1.2.0
 html2text==2018.1.9
-httplib2==0.19.0
-idna==2.8
+httplib2==0.19.1
+idna==2.10
 libsass==0.21.0
 oauthlib==3.2.2
 packaging==20.9
-pdfminer.six==20220319
+pdfminer-six==20211012
 pillow==9.3.0
-protobuf==3.18.3
-psycopg2-binary==2.9.5
-pyasn1-modules==0.2.5
-pyasn1==0.4.5
-pycparser==2.19
-pyopenssl==19.0.0
+proto-plus==1.22.2
+protobuf==3.20.3
+psycopg2==2.9.5
+pyasn1-modules==0.2.8
+pyasn1==0.4.8
+pycparser==2.20
+pyopenssl==19.1.0
 pyparsing==2.4.7
-python-dateutil==2.8.1
-python-decouple==3.1
+python-decouple==3.4
 python-json-logger==2.0.2
 pytz==2022.1
-requests-mock==1.7.0
+requests-mock==1.9.3
 requests-oauthlib==1.3.0
 requests==2.27.1
 rsa==4.7.2
@@ -55,9 +59,9 @@
 setuptools==65.5.1
 six==1.16.0
 sqlparse==0.4.2
-typing-extensions==4.3.0
-uritemplate==3.0.0
-urllib3==1.26.9
+tzdata==2022.7 ; python_version >= "3.8" and python_version < "4.0" and sys_platform == "win32" \
+uritemplate==3.0.1
+urllib3==1.26.5
 uwsgi==2.0.21
 webencodings==0.5.1
 zxcvbn==4.4.28

[kshalot]

Just glancing over it, that seems to highlight that there were some inconsistencies between the dependency files. poetry-lock on dev resolved cachetools to version 4.2.2, while the requirements specify 3.1.1. Both are consistent with the dependency cachetools = ">=2.0.0,<5.0" but inconsistent with each other.

[bemoody]

Yeah, I noticed there are a bunch of things upgraded (which is probably fine) and a couple things downgraded (which might possibly be a problem).

Getting rid of stuff we don't need, like django-compat, is great. (Note though that I don't think the update script will actually remove such packages; this is something we ought to fix.)

I know we've had a lot of weird issues over the years with psycopg2 vs psycopg2-binary. Using psycopg2 is strongly recommended by the authors (https://www.psycopg.org/docs/install.html#psycopg-vs-psycopg-binary).

In the longer term, I'd like to make more dependencies optional (things like psycopg2, uwsgi, and google-api-* are not needed for a demo server) as well as removing stuff from pyproject.toml if it's not a direct dependency.

Also, I would be fine with increasing the minimum python version to 3.9 (in pyproject.toml) which might help remove some of the cruft.