Closed andycui97 closed 1 year ago
My. main query here is how does renaming
LocalPickle
toTmpDirPickle
help in the PR. Most probably I'm missing some context here so a sync up would be helpful!
I renamed LocalPickle
because it actually describes the newly added CWD pickle
flavor. LocalPickle suggests we are using Pickle to a local directory but that doesnt distinguish between using local tmp vs local working directory. I renamed them both so that there is a distinction.
Description
Merges repeated logic in dvc_dag_PythonOperator.jinja into task_function.jinja.
Changes:
Rename localpickle to tmpdirpickle to better match the behavior of the serialization flavor.
Remove stages logic in DVC pipeline writer to directly use TaskDefinitions where possible and avoid intermediate task representation specific to DVC.
Change dvc_dag_PythonOperator.jinja to use TASK_CODE block generated from the task_function template.
Fixes: LIN-736
Type of change
Please delete options that are not relevant.
How Has This Been Tested?