DataBiosphere / azul

Metadata indexer and query service used for AnVIL, HCA, LungMAP, and CGP
Apache License 2.0
7 stars 2 forks source link

Upgrade direct Python dependencies #6511

Open hannes-ucsc opened 1 month ago

hannes-ucsc commented 1 month ago

It's time that we upgrade the direct run- and build-time dependencies.

hannes-ucsc commented 1 month ago

PyCharm has an intention that is very useful for this:

image

The squiggly lines indicate stale dependencies. In general we don't like to update to potentially immature major or even minor releases, so we prefer 1.2.33 over 2.0.0 if there is no 1.3.x. This is a fuzzy science and we have sufficient test coverage. So update to the latest mature release (a release with a high patch number or where the most recent patch release is at least a couple months old) and go backwards if problems occur. If they do, document that fact in a comment above the dependency declaration.

hannes-ucsc commented 1 month ago

I'd like to get the fix in for https://github.com/boto/boto3/issues/3501

hannes-ucsc commented 1 month ago

Also, fix the ordering in the requirements files

Subject: [PATCH] [4/4] Fix: Base image of Azul image is not pinned to digest (#6446)

- Pin base image of Azul and runner images to digest
- Enforce that azul_docker_registry is empty or ends in slash …
- … but remove slash in azul.config.docker_registry
---
Index: requirements.dev.txt
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/requirements.dev.txt b/requirements.dev.txt
--- a/requirements.dev.txt  (revision c020d0de092eeeb19672089ac8ee1d81161fef8f)
+++ b/requirements.dev.txt  (date 1724175603298)
@@ -1,18 +1,19 @@
 atomicwrites==1.4.1
 awscli==1.29.63
-git+https://github.com/DataBiosphere/azul-chalice@1.30.0+12#egg=chalice
+boto3-stubs[s3,sqs,lambda,dynamodb,iam,ecr,stepfunctions,kms]==1.28.63  # match this with the version of the `boto3` runtime dependency
 coverage==6.5.0
 docker==7.1.0
 flake8==5.0.4
 gevent==23.9.1
+git+https://github.com/DataBiosphere/azul-chalice@1.30.0+12#egg=chalice
+git+https://github.com/hannes-ucsc/requirements-parser@v0.2.0+1#egg=requirements-parser
 gitpython==3.1.41
 google-api-python-client==2.103.0
-google-cloud-storage==2.12.0
 google-cloud-bigquery-reservation==1.11.3
+google-cloud-storage==2.12.0
 jq==1.3.0
 locust==2.12.2
 moto[s3,sqs,sns,dynamodb,iam]==4.1.13  # match the extras with the backends listed in AzulUnitTestCase._reset_moto
-boto3-stubs[s3,sqs,lambda,dynamodb,iam,ecr,stepfunctions,kms]==1.28.63  # match this with the version of the `boto3` runtime dependency
 openapi-spec-validator==0.5.1
 openpyxl==3.0.6
 posix_ipc==1.1.1
@@ -20,7 +21,6 @@
 python-dxf==11.4.0
 python-gitlab==3.13.0
 pyyaml==6.0.1
-git+https://github.com/hannes-ucsc/requirements-parser@v0.2.0+1#egg=requirements-parser
 responses==0.22.0
 strict-rfc3339==0.7
 watchdog==2.1.9
Index: requirements.txt
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/requirements.txt b/requirements.txt
--- a/requirements.txt  (revision c020d0de092eeeb19672089ac8ee1d81161fef8f)
+++ b/requirements.txt  (date 1724175622136)
@@ -9,9 +9,9 @@
 elasticsearch==7.17.9
 fastavro==1.8.2
 furl==2.1.3
+google-api-core==2.12.0
 google-auth[pyopenssl]==2.23.3
 google-cloud-bigquery==3.12.0
-google-api-core==2.12.0
 http-message-signatures==0.4.4
 jmespath==1.0.1
 more-itertools==9.0.0