search

LineaLabs / lineapy

Move fast from data science prototype to pipeline. Capture, analyze, and transform messy notebooks into data pipelines with just two lines of code.
https://lineapy.org
Apache License 2.0
664 stars 58 forks source link

DVC Split Module Files #871

Open andycui97 opened 1 year ago

andycui97 commented 1 year ago

Currently DVC pipelines use the same module file and have multiple task files calling functions from the module file.

This is not desired behavior as it causes the DVC cache to recompute all the tasks if any changes are made to the shared module file.

We should split the module file up. Possible approaches may be to write the module file's functions directly in the corresponding task file.

andycui97 commented 1 year ago

cc: @pd-t ,

DVC implemented in : https://github.com/LineaLabs/lineapy/issues/765