astronomer / astro-provider-snowflake

Apache License 2.0
0 stars 0 forks source link

Refactor backend runner to not require airflow #7

Closed mpgreg closed 1 year ago

mpgreg commented 1 year ago

Example backend runner is at https://github.com/mpgreg/airflow-snowpark-containers-demo/tree/main/include/runner. This code uses airflow and runs the python_callable using the PythonVirtualenvOperator and a new customer xcom backend (https://github.com/astronomer/astro-provider-snowflake/blob/main/provider/astronomer/providers/snowflake/xcom_backends/snowflake.py) for passing data to/from the task.

This needs to be refactored to not require Airflow. Can reuse the code at https://github.com/apache/airflow/blob/7ebda3898db2eee72d043a9565a674dea72cd8fa/airflow/utils/python_virtualenv.py#L63 and https://github.com/apache/airflow/blob/7ebda3898db2eee72d043a9565a674dea72cd8fa/airflow/utils/python_virtualenv.py#L104 to create the environment.

Challenges will be with the serialization and deserialization of task I/O. Will need to reevaluate how to use logic from https://github.com/astronomer/astro-provider-snowflake/blob/main/provider/astronomer/providers/snowflake/xcom_backends/snowflake.py

mpgreg commented 1 year ago

Completed

mpgreg commented 1 year ago

completed