Parametrize and refactor inter task communication blocks.
Major changes:
Move get_artifact_task_definitions and get_session_task_definitions to task_gen.py, and hide them behind a generic get_task_definition that takes in DagTaskBreakdown instance as parameter.
Move all airflow specific configs to lineapy/plugins/airflow_pipeline_writer.py and DVC to lineapy/plugins/dvc_pipeline_writer.py
Add TaskSerializer enum and render_task_io_serialize_blocks which render task serialization blocks and can be parametrized through dag configs, this config added to Airflow as task_serialization.
Add get_localpickle_setup_task_definition, and get_localpickle_teardown_task_definition, which returns a Task for setting up local pickle directory and tearing it down.
Minor changes
Add comments to all new functions
Move extract_taskgraph from lineapy/api/api_utils.py to lineapy/plugins/task.py to avoid circular dep
Fixes: LIN-625
Type of change
Please delete options that are not relevant.
[X] Refactor (non-breaking change)
[X] This change requires a documentation update
How Has This Been Tested?
Existing pytests. Updated snapshots but verified airflow frameworks still run with airflow marked pytests.
Description
Parametrize and refactor inter task communication blocks.
Major changes:
get_artifact_task_definitions
andget_session_task_definitions
totask_gen.py
, and hide them behind a genericget_task_definition
that takes inDagTaskBreakdown
instance as parameter.lineapy/plugins/airflow_pipeline_writer.py
and DVC tolineapy/plugins/dvc_pipeline_writer.py
render_task_io_serialize_blocks
which render task serialization blocks and can be parametrized through dag configs, this config added to Airflow astask_serialization
.get_localpickle_setup_task_definition
, andget_localpickle_teardown_task_definition
, which returns a Task for setting up local pickle directory and tearing it down.Minor changes
extract_taskgraph
fromlineapy/api/api_utils.py
tolineapy/plugins/task.py
to avoid circular depFixes: LIN-625
Type of change
Please delete options that are not relevant.
How Has This Been Tested?
Existing pytests. Updated snapshots but verified airflow frameworks still run with airflow marked pytests.