search

GoogleCloudPlatform / cloud-sql-python-connector

A Python library for connecting securely to your Cloud SQL instances.
Apache License 2.0
278 stars 65 forks source link

Create an Official Beam I/O Connector for Cloud SQL #1062

Open FearTheMonstar opened 4 months ago

FearTheMonstar commented 4 months ago

Feature Description

The current Cloud SQL Python connector is great. But in it's current form, when used via Apache Beam/Google Cloud Dataflow, it is not recognized as an official Apache Beam framework construct. This can lead to unbalanced workers in Dataflow when dealing with any kind of volume in a pipeline.

I suggest creating an official Beam I/O Connector so that Cloud SQL can have optimized usage with Beam/Dataflow.

Sample code

No response

Alternatives Considered

There is current non-native Cloud SQL support, but it requires a custom container. This can cause slowdowns and longer approval processes at regulated customers. An official native I/O connector would be ideal.

Additional Details

No response

jackwotherspoon commented 4 months ago

Thanks for raising the issue! We will chat with stakeholders and monitor any upvotes here to help prioritize this potential work.

Related to #1057

danvau7 commented 4 months ago

Would like to add 1 upvote to this item :)