LineaLabs / lineapy

Move fast from data science prototype to pipeline. Capture, analyze, and transform messy notebooks into data pipelines with just two lines of code.
https://lineapy.org
Apache License 2.0
664 stars 58 forks source link

LIN-736: Merge dvc_dag_PythonOperator.jinja into task_function.jinja #872

Closed andycui97 closed 1 year ago

andycui97 commented 1 year ago

Description

Merges repeated logic in dvc_dag_PythonOperator.jinja into task_function.jinja.

Changes:

Fixes: LIN-736

Type of change

Please delete options that are not relevant.

How Has This Been Tested?

andycui97 commented 1 year ago

My. main query here is how does renaming LocalPickle to TmpDirPickle help in the PR. Most probably I'm missing some context here so a sync up would be helpful!

I renamed LocalPickle because it actually describes the newly added CWD pickle flavor. LocalPickle suggests we are using Pickle to a local directory but that doesnt distinguish between using local tmp vs local working directory. I renamed them both so that there is a distinction.