Closed mpgreg closed 1 year ago
Example backend runner is at https://github.com/mpgreg/airflow-snowpark-containers-demo/tree/main/include/runner. This code uses airflow and runs the python_callable using the PythonVirtualenvOperator and a new customer xcom backend (https://github.com/astronomer/astro-provider-snowflake/blob/main/provider/astronomer/providers/snowflake/xcom_backends/snowflake.py) for passing data to/from the task.
This needs to be refactored to not require Airflow. Can reuse the code at https://github.com/apache/airflow/blob/7ebda3898db2eee72d043a9565a674dea72cd8fa/airflow/utils/python_virtualenv.py#L63 and https://github.com/apache/airflow/blob/7ebda3898db2eee72d043a9565a674dea72cd8fa/airflow/utils/python_virtualenv.py#L104 to create the environment.
Challenges will be with the serialization and deserialization of task I/O. Will need to reevaluate how to use logic from https://github.com/astronomer/astro-provider-snowflake/blob/main/provider/astronomer/providers/snowflake/xcom_backends/snowflake.py
Completed
completed
Example backend runner is at https://github.com/mpgreg/airflow-snowpark-containers-demo/tree/main/include/runner. This code uses airflow and runs the python_callable using the PythonVirtualenvOperator and a new customer xcom backend (https://github.com/astronomer/astro-provider-snowflake/blob/main/provider/astronomer/providers/snowflake/xcom_backends/snowflake.py) for passing data to/from the task.
This needs to be refactored to not require Airflow. Can reuse the code at https://github.com/apache/airflow/blob/7ebda3898db2eee72d043a9565a674dea72cd8fa/airflow/utils/python_virtualenv.py#L63 and https://github.com/apache/airflow/blob/7ebda3898db2eee72d043a9565a674dea72cd8fa/airflow/utils/python_virtualenv.py#L104 to create the environment.
Challenges will be with the serialization and deserialization of task I/O. Will need to reevaluate how to use logic from https://github.com/astronomer/astro-provider-snowflake/blob/main/provider/astronomer/providers/snowflake/xcom_backends/snowflake.py