search

GoogleCloudPlatform / cloud-sql-python-connector

A Python library for connecting securely to your Cloud SQL instances.
Apache License 2.0
277 stars 65 forks source link

GCP Composer unable to install cloud-sql-python-connector pypi package #1082

Closed software1tkh closed 4 days ago

software1tkh commented 4 months ago

Bug Description

Tried to install latest cloud-sql-python-connector pypi package into GCP Composer but failed.
Tried it on both images:
composer-2.6.1-airflow-2.6.3
composer-2.7.1-airflow-2.7.3

The following cloud-sql-python-connector pypi package version installed successfully: 0.6.2 0.7.0 0.9.0 1.1.0 1.4.0 1.6.0

Anything higher than 1.6.0 and lower than 0.6.2 will fail.

Error logs caused this issue : There was an error creating your Issue: body is too long (maximum is 65536 characters). So paste the logs in https://pastebin.com/k5Pa8tpW instead

Example code (or command)

No response

Stacktrace

https://pastebin.com/k5Pa8tpW

Steps to reproduce?

  1. Login into GCP web console -> GCP Composer -> Pypi Package -> Edit -> key in the following: Package Name 1 : cloud-sql-python-connector Extras and Version 1 : ==1.9.2

Environment

  1. OS type and version: composer-2.6.1-airflow-2.6.3 & composer-2.7.1-airflow-2.7.3
  2. Python version: 3.11.5 & 3.11.8
  3. Cloud SQL Python Connector version: 1.70, 1.8.0, 1.9.0, 1.9.1, 1.9.2

Additional Details

No response

jackwotherspoon commented 3 months ago

Hi @software1tkh, thanks for raising an issue on the Cloud SQL Python Connector 😄

I took a quick peek into this and am wondering if the issue is maybe because of a dependency conflict between a pre-installed dependency on the composer image and the same package dependency with a different version from the Cloud SQL Python Connector.

I took a look at the composer version dependencies https://cloud.google.com/composer/docs/concepts/versioning/composer-versions and it seems the composer image in question has cryptography==41.0.7 while the Python Connector in version 1.7.0 onwards requires cryptography>=42.0.0.

This is just a hunch. If it is the cause then the path forward would be to for me to get the composer folks to bump their dependency versions to increase the pre-installed version to above 42.0.0 for the cryptography package.

I will give a go at reproducing this and see if I can find a solution.

While it is not ideal, I'd recommend using v1.6.0 of the Python Connector in the meantime.

jackwotherspoon commented 2 weeks ago

Just putting an update here. In the next composer version v2.9.3, the cryptography package should be bumped up to cryptography==43.0.0. Will monitor and post here when version is released.

jackwotherspoon commented 4 days ago

Latest release on Sept 2, 2024 composer-3-airflow-2.9.1-build.6 can now be found on the Cloud Composer Versions page and includes cryptography==43.0.0.

This unblocks all latest releases of the Cloud SQL Python Connector being used with Composer 👏 😄