Add custom XCOM backends - GCS

astronomer / astronomer-providers

Airflow Providers containing Deferrable Operators & Sensors from Astronomer

https://astronomer-providers.rtfd.io/

Apache License 2.0

137 stars 26 forks source link

Closed kaxil closed 1 year ago

kaxil commented 2 years ago

Refer to https://astronomer.slack.com/archives/CGQSYG25V/p1658957159758039 for the relevant discussion, and add GCS XCOM Backend into the Google Provider

rajaths010494 commented 2 years ago

Unit test for Pandas Dataframe in xcom is pending.

rajaths010494 commented 2 years ago

Googler test cases were failing because of protobuf <3.20 . Fixed it and other tests are running fine now.

rajaths010494 commented 2 years ago

fixed review comments. upload non listed datatypes as bytes to download review is pending.

phanikumv commented 2 years ago

Test it with astro-sdk by setting custom XCM backend

rajaths010494 commented 2 years ago

trying astro-sdk to test and pyarrow to explore more datatypes.

phanikumv commented 2 years ago