KubernetesPodOperator prints empty log lines without respecting the log format

karakanb commented 5 months ago

Apache Airflow Provider(s)

cncf-kubernetes

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==8.14.0 apache-airflow-providers-celery==3.5.1 apache-airflow-providers-cncf-kubernetes==7.12.0 apache-airflow-providers-common-io==1.1.0 apache-airflow-providers-common-sql==1.10.0 apache-airflow-providers-discord==3.5.0 apache-airflow-providers-docker==3.8.2 apache-airflow-providers-elasticsearch==5.3.0 apache-airflow-providers-ftp==3.7.0 apache-airflow-providers-google==10.13.0 apache-airflow-providers-grpc==3.4.0 apache-airflow-providers-hashicorp==3.6.0 apache-airflow-providers-http==4.8.0 apache-airflow-providers-imap==3.5.0 apache-airflow-providers-microsoft-azure==8.4.0 apache-airflow-providers-mysql==5.5.0 apache-airflow-providers-odbc==4.2.0 apache-airflow-providers-openlineage==1.3.0 apache-airflow-providers-postgres==5.9.0 apache-airflow-providers-redis==3.5.0 apache-airflow-providers-sendgrid==3.4.0 apache-airflow-providers-sftp==4.8.0 apache-airflow-providers-slack==8.5.1 apache-airflow-providers-snowflake==5.2.1 apache-airflow-providers-sqlite==3.6.0 apache-airflow-providers-ssh==3.9.0

Apache Airflow version

2.8.0

Operating System

Debian 12 (bookworm)

Deployment

Official Apache Airflow Helm Chart

Deployment details

deployed on digitalocean k8s. I have a custom log format as '%%(asctime)s | %%(levelname)s | %%(message)s'

What happened

I have recently upgraded the cncf-kubernetes provider from 7.4.2 to 7.12.0 and started seeing weird log lines being printed out in-between application logs:

2024-01-03T21:31:59.286+0000 | WARNING | Pod not yet started: my-pod-name-xutbmgq3
2024-01-03T21:32:00.313+0000 | WARNING | Pod not yet started: my-pod-name-xutbmgq3
2024-01-03T21:32:00.694053856Z 
2024-01-03T21:32:00.694107912Z 
2024-01-03T21:32:00.694384093Z 
2024-01-03T21:32:00.694416033Z 
2024-01-03T21:32:00.694514264Z 
2024-01-03T21:32:00.694543915Z 
2024-01-03T21:32:00.694553314Z 
2024-01-03T21:32:00.694557782Z 
2024-01-03T21:32:00.694629859Z 
2024-01-03T21:32:00.694642682Z 
2024-01-03T21:32:01.348+0000 | WARNING | Pod not yet started: my-pod-name-xutbmgq3

this also happens while the application is running:

2024-01-03T21:32:04.814+0000 | INFO | [base] application log line 1
2024-01-03T21:32:05.234+0000 | INFO | [base] application log line 2
2024-01-03T21:32:05.526911055Z 
2024-01-03T21:32:05.529+0000 | INFO | [base] application log line 3
2024-01-03T21:32:05.529+0000 | INFO | [base] application log line 4
2024-01-03T21:32:05.654767914Z 
2024-01-03T21:32:05.654830820Z 
2024-01-03T21:32:05.654857369Z

As you can see the log lines have a different format than the actual airflow log format, and they seem to contain no messages. This causes confusion, and for a few cases where I parse the logs it broke my existing code.

I have checked through external log collectors as well as via tailing the pod logs and I can confirm that the logs do not come from the application/pod side, and being logged somewhere on the airflow side.

What you think should happen instead

there should be no log lines printed that doesn't respect the logging format.

How to reproduce

Create a pod with KubernetesPodOperator, it happens to me consistently.

Anything else

Every time I run a pod it happens.

Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

hussein-awala commented 5 months ago

This log is very strange; for example, for the pod starting wait, the check is running in a loop, and there are no other log statements: https://github.com/apache/airflow/blob/2c15dc9dd1b360bcb3b6d6c2e41e50ef0d97801e/airflow/providers/cncf/kubernetes/utils/pod_manager.py#L355-L366

These log messages come from something else, probably from the TaskContextLogger since you are using Airflow 2.8.0. Could you try to deactivate the feature and test the KPO? https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#enable-task-context-logger

vatsrahul1001 commented 5 months ago

Agree log is very strange, @karakanb I am not able to replicate this with 2.8.0 & cncf-kubernetes==7.12.0, kindly do update us if you do not see this issue after disabling TaskContextLogger

[2024-01-04, 09:15:41 UTC] {pod_manager.py:359} WARNING - Pod not yet started: passing-test-o9zwtu7g [2024-01-04, 09:15:42 UTC] {pod_manager.py:359} WARNING - Pod not yet started: passing-test-o9zwtu7g [2024-01-04, 09:15:43 UTC] {pod_manager.py:359} WARNING - Pod not yet started: passing-test-o9zwtu7g [2024-01-04, 09:15:44 UTC] {pod_manager.py:359} WARNING - Pod not yet started: passing-test-o9zwtu7g [2024-01-04, 09:15:45 UTC] {pod_manager.py:359} WARNING - Pod not yet started: passing-test-o9zwtu7g [2024-01-04, 09:15:46 UTC] {pod_manager.py:359} WARNING - Pod not yet started: passing-test-o9zwtu7g [2024-01-04, 09:15:47 UTC] {pod_manager.py:359} WARNING - Pod not yet started: passing-test-o9zwtu7g [2024-01-04, 09:15:48 UTC] {pod_manager.py:359} WARNING - Pod not yet started: passing-test-o9zwtu7g [2024-01-04, 09:15:49 UTC] {pod_manager.py:359} WARNING - Pod not yet started: passing-test-o9zwtu7g [2024-01-04, 09:15:50 UTC] {pod_manager.py:359} WARNING - Pod not yet started: passing-test-o9zwtu7g [2024-01-04, 09:15:51 UTC] {pod_manager.py:359} WARNING - Pod not yet started: passing-test-o9zwtu7g [2024-01-04, 09:15:52 UTC] {pod_manager.py:359} WARNING - Pod not yet started: passing-test-o9zwtu7g [2024-01-04, 09:15:53 UTC] {pod_manager.py:359} WARNING - Pod not yet started: passing-test-o9zwtu7g [2024-01-04, 09:15:54 UTC] {pod_manager.py:359} WARNING - Pod not yet started: passing-test-o9zwtu7g [2024-01-04, 09:15:55 UTC] {pod_manager.py:359} WARNING - Pod not yet started: passing-test-o9zwtu7g [2024-01-04, 09:15:56 UTC] {pod_manager.py:359} WARNING - Pod not yet started: passing-test-o9zwtu7g [2024-01-04, 09:15:57 UTC] {pod_manager.py:359} WARNING - Pod not yet started: passing-test-o9zwtu7g [2024-01-04, 09:15:58 UTC] {pod_manager.py:460} INFO - [base] hello world [2024-01-04, 09:15:58 UTC] {pod_manager.py:488} WARNING - Pod passing-test-o9zwtu7g log read interrupted but container base still running [2024-01-04, 09:15:59 UTC] {pod_manager.py:460} INFO - [base] hello world [2024-01-04, 09:15:59 UTC] {pod.py:848} INFO - Deleting pod: passing-test-o9zwtu7g [2024-01-04, 09:15:59 UTC] {taskinstance.py:1138} INFO - Marking task as SUCCESS. dag_id=k8pod_operator, task_id=passing-task, execution_date=20240104T091538, start_date=20240104T091541, end_date=20240104T091559 [2024-01-04, 09:15:59 UTC] {local_task_job_runner.py:234} INFO - Task exited with return code 0 [2024-01-04, 09:15:59 UTC] {taskinstance.py:3281} INFO - 0 downstream tasks scheduled from follow-on schedule check

karakanb commented 5 months ago

I can confirm that the issue is still there even after I disabled the context logger:

[logging]
colored_console_log = False
enable_task_context_logger = False
log_format = %%(asctime)s | %%(levelname)s | %%(message)s
remote_logging = False

vatsrahul1001 commented 5 months ago

@karakanb, since you've upgraded the CNCF Kubernetes provider from version 7.4.2 to 7.12.0, does that imply you weren't experiencing the issue with 7.4.2 and only observed it after the upgrade to 7.12.0?

karakanb commented 5 months ago

Exactly, the issue wasn't happening in 7.4.2, it has started happening in 7.12.0, although I have updated pretty much all the other providers as well as the airflow version at the same time. Let me test it by downgrading the cncf provider.

karakanb commented 5 months ago

The issue disappeared when I went back to 7.4.2, which means it has nothing to do with Airflow 2.8.0 or my environment as far as I can tell. I have upgraded to v7.13.0 and the issue persists there as well.

potiuk commented 5 months ago

cc: @dstandish - likely #35416 was not THAT inconsequential

hussein-awala commented 5 months ago

@potiuk according to the provided log:

2024-01-03T21:31:59.286+0000 | WARNING | Pod not yet started: my-pod-name-xutbmgq3
2024-01-03T21:32:00.313+0000 | WARNING | Pod not yet started: my-pod-name-xutbmgq3
2024-01-03T21:32:00.694053856Z 
2024-01-03T21:32:00.694107912Z 
2024-01-03T21:32:00.694384093Z 
2024-01-03T21:32:00.694416033Z 
2024-01-03T21:32:00.694514264Z 
2024-01-03T21:32:00.694543915Z 
2024-01-03T21:32:00.694553314Z 
2024-01-03T21:32:00.694557782Z 
2024-01-03T21:32:00.694629859Z 
2024-01-03T21:32:00.694642682Z 
2024-01-03T21:32:01.348+0000 | WARNING | Pod not yet started: my-pod-name-xutbmgq3

the empty log messages are related to this loop: https://github.com/apache/airflow/blob/2c15dc9dd1b360bcb3b6d6c2e41e50ef0d97801e/airflow/providers/cncf/kubernetes/utils/pod_manager.py#L355-L366

and most probably logged by a 3rd party lib (tenacity.retry or kubernetes read_namespaced_pod methods) 🤔

vatsrahul1001 commented 5 months ago

@potiuk I also tried reproducing this as mentioned here and not able to repro with cncf 7.12.0, I also suspects it is being logged by some 3rd party lib.

potiuk commented 5 months ago

Yeah. It's just strange changing provider version fixes it - but yeah it could be caused also by some changes to provider dependencies between those two versions (but it does not look like it could be the reason):

git diff providers-cncf-kubernetes/7.4.2..providers-cncf-kubernetes/7.12.0 -- airflow/providers/cncf/kubernetes/provider.yaml

diff --git a/airflow/providers/cncf/kubernetes/provider.yaml b/airflow/providers/cncf/kubernetes/provider.yaml
index 11a661e63e..f0b35ca392 100644
--- a/airflow/providers/cncf/kubernetes/provider.yaml
+++ b/airflow/providers/cncf/kubernetes/provider.yaml
@@ -22,7 +22,17 @@ description: |
     `Kubernetes <https://kubernetes.io/>`__

 suspended: false
+source-date-epoch: 1703288121
 versions:
+  - 7.12.0
+  - 7.11.0
+  - 7.10.0
+  - 7.9.0
+  - 7.8.0
+  - 7.7.0
+  - 7.6.0
+  - 7.5.1
+  - 7.5.0
   - 7.4.2
   - 7.4.1
   - 7.4.0
@@ -65,7 +75,8 @@ versions:
   - 1.0.0

 dependencies:
-  - apache-airflow>=2.4.0
+  - aiofiles>=23.2.0
+  - apache-airflow>=2.6.0
   - asgiref>=3.5.2
   - cryptography>=2.0.0
   # The Kubernetes API is known to introduce problems when upgraded to a MAJOR version. Airflow Core
@@ -100,7 +111,6 @@ integrations:
 operators:
   - integration-name: Kubernetes
     python-modules:
-      - airflow.providers.cncf.kubernetes.operators.kubernetes_pod
       - airflow.providers.cncf.kubernetes.operators.pod
       - airflow.providers.cncf.kubernetes.operators.spark_kubernetes
       - airflow.providers.cncf.kubernetes.operators.resource
@@ -337,3 +347,6 @@ config:
         type: string
         example: ~
         default: ""
+
+executors:
+  - airflow.providers.cncf.kubernetes.kubernetes_executor.KubernetesExecutor

karakanb commented 5 months ago

Here's the dockerfile I use for this deployment:

FROM apache/airflow:2.8.0-python3.11

ENV PYTHONPATH "${PYTHONPATH}:${AIRFLOW_HOME}"
ENV SQLALCHEMY_SILENCE_UBER_WARNING=1

USER root
RUN apt-get update && apt-get install -y git gcc g++ unixodbc unixodbc-dev make nano curl

USER airflow

COPY requirements.txt /
RUN pip install --no-cache-dir "apache-airflow==${AIRFLOW_VERSION}" -r /requirements.txt

COPY . .

The requirements.txt:

apache-airflow-providers-amazon==8.14.0
apache-airflow-providers-celery==3.5.1
apache-airflow-providers-cncf-kubernetes==7.13.0
apache-airflow-providers-common-sql==1.10.0
apache-airflow-providers-discord==3.5.0
apache-airflow-providers-google==10.13.0
apache-airflow-providers-slack==8.5.1
apache-airflow-providers-snowflake==5.2.1
apache-airflow===2.8.0
deepdiff==6.7.1
isort==5.13.2
mypy==1.8.0
pendulum==2.1.2
pydantic==2.5.3
pytest-cov==4.1.0
pytest==7.4.3
pyyaml==6.0.1 
types-PyYAML==6.0.1
requests-mock==1.11.0
ruff==0.1.9
tabulate==0.9.0
hvac==2.0.0

The full set of dependencies, in case it helps:

``` adal==1.2.7 adlfs==2023.10.0 aiobotocore==2.9.0 aiofiles==23.2.1 aiohttp==3.8.6 aioitertools==0.11.0 aiosignal==1.3.1 alembic==1.13.0 amqp==5.2.0 annotated-types==0.6.0 anyio==4.1.0 apache-airflow==2.8.0 apache-airflow-providers-amazon==8.14.0 apache-airflow-providers-celery==3.5.1 apache-airflow-providers-cncf-kubernetes==7.13.0 apache-airflow-providers-common-io==1.1.0 apache-airflow-providers-common-sql==1.10.0 apache-airflow-providers-discord==3.5.0 apache-airflow-providers-docker==3.8.2 apache-airflow-providers-elasticsearch==5.3.0 apache-airflow-providers-ftp==3.7.0 apache-airflow-providers-google==10.13.0 apache-airflow-providers-grpc==3.4.0 apache-airflow-providers-hashicorp==3.6.0 apache-airflow-providers-http==4.8.0 apache-airflow-providers-imap==3.5.0 apache-airflow-providers-microsoft-azure==8.4.0 apache-airflow-providers-mysql==5.5.0 apache-airflow-providers-odbc==4.2.0 apache-airflow-providers-openlineage==1.3.0 apache-airflow-providers-postgres==5.9.0 apache-airflow-providers-redis==3.5.0 apache-airflow-providers-sendgrid==3.4.0 apache-airflow-providers-sftp==4.8.0 apache-airflow-providers-slack==8.5.1 apache-airflow-providers-snowflake==5.2.1 apache-airflow-providers-sqlite==3.6.0 apache-airflow-providers-ssh==3.9.0 apispec==6.3.0 argcomplete==3.2.1 asgiref==3.7.2 asn1crypto==1.5.1 async-timeout==4.0.3 attrs==23.1.0 Authlib==1.2.1 azure-batch==14.0.0 azure-common==1.1.28 azure-core==1.29.6 azure-cosmos==4.5.1 azure-datalake-store==0.0.53 azure-identity==1.15.0 azure-keyvault-secrets==4.7.0 azure-kusto-data==4.3.0 azure-mgmt-containerinstance==10.1.0 azure-mgmt-containerregistry==10.3.0 azure-mgmt-core==1.4.0 azure-mgmt-cosmosdb==9.3.0 azure-mgmt-datafactory==4.0.0 azure-mgmt-datalake-nspkg==3.0.1 azure-mgmt-datalake-store==0.5.0 azure-mgmt-nspkg==3.0.2 azure-mgmt-resource==23.0.1 azure-mgmt-storage==21.1.0 azure-nspkg==3.0.2 azure-servicebus==7.11.4 azure-storage-blob==12.19.0 azure-storage-file-datalake==12.14.0 azure-storage-file-share==12.15.0 azure-synapse-artifacts==0.18.0 azure-synapse-spark==0.7.0 Babel==2.14.0 backoff==2.2.1 bcrypt==4.1.2 beautifulsoup4==4.12.2 billiard==4.2.0 blinker==1.7.0 boto3==1.33.13 botocore==1.33.13 cachelib==0.9.0 cachetools==5.3.2 cattrs==23.2.3 celery==5.3.6 certifi==2023.11.17 cffi==1.16.0 chardet==5.2.0 charset-normalizer==3.3.2 click==8.1.7 click-didyoumean==0.3.0 click-plugins==1.1.1 click-repl==0.3.0 clickclick==20.10.2 colorama==0.4.6 colorlog==4.8.0 ConfigUpdater==3.2 connexion==2.14.2 coverage==7.4.0 cron-descriptor==1.4.0 croniter==2.0.1 cryptography==41.0.7 db-dtypes==1.2.0 decorator==5.1.1 deepdiff==6.7.1 Deprecated==1.2.14 dill==0.3.1.1 distlib==0.3.8 dnspython==2.4.2 docker==7.0.0 docutils==0.20.1 elastic-transport==8.11.0 elasticsearch==8.11.1 email-validator==1.3.1 eventlet==0.33.3 filelock==3.13.1 Flask==2.2.5 Flask-AppBuilder==4.3.10 Flask-Babel==2.0.0 Flask-Caching==2.1.0 Flask-JWT-Extended==4.6.0 Flask-Limiter==3.5.0 Flask-Login==0.6.3 Flask-Session==0.5.0 Flask-SQLAlchemy==2.5.1 Flask-WTF==1.2.1 flower==2.0.1 frozenlist==1.4.1 fsspec==2023.12.2 gcloud-aio-auth==4.2.3 gcloud-aio-bigquery==7.0.0 gcloud-aio-storage==9.0.0 gcsfs==2023.12.2.post1 gevent==23.9.1 google-ads==22.1.0 google-api-core==2.15.0 google-api-python-client==2.111.0 google-auth==2.25.2 google-auth-httplib2==0.2.0 google-auth-oauthlib==1.2.0 google-cloud-aiplatform==1.38.1 google-cloud-appengine-logging==1.4.0 google-cloud-audit-log==0.2.5 google-cloud-automl==2.12.0 google-cloud-batch==0.17.6 google-cloud-bigquery==3.14.1 google-cloud-bigquery-datatransfer==3.13.0 google-cloud-bigquery-storage==2.24.0 google-cloud-bigtable==2.22.0 google-cloud-build==3.22.0 google-cloud-compute==1.15.0 google-cloud-container==2.36.0 google-cloud-core==2.4.1 google-cloud-datacatalog==3.17.0 google-cloud-dataflow-client==0.8.6 google-cloud-dataform==0.5.5 google-cloud-dataplex==1.10.0 google-cloud-dataproc==5.8.0 google-cloud-dataproc-metastore==1.14.0 google-cloud-dlp==3.14.0 google-cloud-kms==2.20.0 google-cloud-language==2.12.0 google-cloud-logging==3.9.0 google-cloud-memcache==1.8.0 google-cloud-monitoring==2.18.0 google-cloud-orchestration-airflow==1.10.0 google-cloud-os-login==2.12.0 google-cloud-pubsub==2.19.0 google-cloud-redis==2.14.0 google-cloud-resource-manager==1.11.0 google-cloud-run==0.10.1 google-cloud-secret-manager==2.17.0 google-cloud-spanner==3.40.1 google-cloud-speech==2.23.0 google-cloud-storage==2.14.0 google-cloud-storage-transfer==1.10.0 google-cloud-tasks==2.15.0 google-cloud-texttospeech==2.15.0 google-cloud-translate==3.13.0 google-cloud-videointelligence==2.12.0 google-cloud-vision==3.5.0 google-cloud-workflows==1.13.0 google-crc32c==1.5.0 google-re2==1.1 google-resumable-media==2.7.0 googleapis-common-protos==1.62.0 graphviz==0.20.1 greenlet==3.0.2 grpc-google-iam-v1==0.13.0 grpcio==1.60.0 grpcio-gcp==0.2.2 grpcio-status==1.60.0 gunicorn==21.2.0 h11==0.14.0 httpcore==0.16.3 httplib2==0.22.0 httpx==0.23.3 humanize==4.9.0 hvac==2.0.0 idna==3.6 ijson==3.2.3 importlib-metadata==6.11.0 importlib-resources==6.1.1 inflection==0.5.1 iniconfig==2.0.0 isodate==0.6.1 isort==5.13.2 itsdangerous==2.1.2 Jinja2==3.1.2 jmespath==0.10.0 json-merge-patch==0.2 jsonpath-ng==1.6.0 jsonschema==4.20.0 jsonschema-specifications==2023.11.2 kombu==5.3.4 kubernetes==23.6.0 kubernetes-asyncio==24.2.3 lazy-object-proxy==1.10.0 ldap3==2.9.1 limits==3.7.0 linkify-it-py==2.0.2 lockfile==0.12.2 looker-sdk==23.20.0 lxml==4.9.3 Mako==1.3.0 Markdown==3.5.1 markdown-it-py==3.0.0 MarkupSafe==2.1.3 marshmallow==3.20.1 marshmallow-oneofschema==3.0.1 marshmallow-sqlalchemy==0.26.1 mdit-py-plugins==0.4.0 mdurl==0.1.2 msal==1.26.0 msal-extensions==1.1.0 msrest==0.7.1 msrestazure==0.6.4 multidict==6.0.4 mypy==1.8.0 mypy-extensions==1.0.0 mysql-connector-python==8.0.29 mysqlclient==2.2.1 numpy==1.24.4 oauthlib==3.2.2 openlineage-integration-common==1.6.2 openlineage-python==1.6.2 openlineage_sql==1.6.2 opentelemetry-api==1.22.0 opentelemetry-exporter-otlp==1.22.0 opentelemetry-exporter-otlp-proto-common==1.22.0 opentelemetry-exporter-otlp-proto-grpc==1.22.0 opentelemetry-exporter-otlp-proto-http==1.22.0 opentelemetry-proto==1.22.0 opentelemetry-sdk==1.22.0 opentelemetry-semantic-conventions==0.43b0 ordered-set==4.1.0 packaging==23.2 pandas==2.1.4 pandas-gbq==0.20.0 paramiko==3.3.1 pathspec==0.12.1 pendulum==2.1.2 platformdirs==3.11.0 pluggy==1.3.0 ply==3.11 portalocker==2.8.2 prison==0.2.1 prometheus-client==0.19.0 prompt-toolkit==3.0.43 proto-plus==1.23.0 protobuf==4.25.1 psutil==5.9.6 psycopg2-binary==2.9.9 pyarrow==11.0.0 pyarrow-hotfix==0.6 pyasn1==0.5.1 pyasn1-modules==0.3.0 pycparser==2.21 pydantic==2.5.3 pydantic_core==2.14.6 pydata-google-auth==1.8.2 Pygments==2.17.2 PyJWT==2.8.0 PyNaCl==1.5.0 pyodbc==5.0.1 pyOpenSSL==23.3.0 pyparsing==3.1.1 pytest==7.4.3 pytest-cov==4.1.0 python-daemon==3.0.1 python-dateutil==2.8.2 python-dotenv==1.0.0 python-http-client==3.3.7 python-ldap==3.4.4 python-nvd3==0.15.0 python-slugify==8.0.1 pytz==2023.3.post1 pytzdata==2020.1 PyYAML==6.0.1 redis==4.6.0 redshift-connector==2.0.918 referencing==0.32.0 requests==2.31.0 requests-mock==1.11.0 requests-oauthlib==1.3.1 requests-toolbelt==1.0.0 rfc3339-validator==0.1.4 rfc3986==1.5.0 rich==13.7.0 rich-argparse==1.4.0 rpds-py==0.13.2 rsa==4.9 ruff==0.1.9 s3transfer==0.8.2 scramp==1.4.4 sendgrid==6.11.0 setproctitle==1.3.3 shapely==2.0.2 six==1.16.0 slack-sdk==3.26.1 sniffio==1.3.0 snowflake-connector-python==3.6.0 snowflake-sqlalchemy==1.5.1 sortedcontainers==2.4.0 soupsieve==2.5 SQLAlchemy==1.4.50 sqlalchemy-bigquery==1.9.0 SQLAlchemy-JSONField==1.0.2 sqlalchemy-redshift==0.8.14 sqlalchemy-spanner==1.6.2 SQLAlchemy-Utils==0.41.1 sqlparse==0.4.4 sshtunnel==0.4.0 starkbank-ecdsa==2.2.0 statsd==4.0.1 tabulate==0.9.0 tenacity==8.2.3 termcolor==2.4.0 text-unidecode==1.3 tomlkit==0.12.3 tornado==6.4 types-PyYAML==6.0.1 typing_extensions==4.9.0 tzdata==2023.3 uc-micro-py==1.0.2 unicodecsv==0.14.1 universal-pathlib==0.1.4 uritemplate==4.1.1 urllib3==2.0.6 vine==5.1.0 virtualenv==20.25.0 watchtower==3.0.1 wcwidth==0.2.12 websocket-client==1.7.0 Werkzeug==2.2.3 wrapt==1.16.0 WTForms==3.1.1 yarl==1.9.4 zipp==3.17.0 zope.event==5.0 zope.interface==6.1 ```