LineaLabs / lineapy

Move fast from data science prototype to pipeline. Capture, analyze, and transform messy notebooks into data pipelines with just two lines of code.
https://lineapy.org
Apache License 2.0
662 stars 58 forks source link

LIN-592 Update NodeCollection dependencies with reuse_pre_calculated_artifact… #809

Closed mingjerli closed 1 year ago

mingjerli commented 1 year ago

Description

When specify dependencies and reuse_pre_computed_artifacts, we get error.

The root cause is we forget to update the artifact calculation dependency graph in the case of reuse_pre_computed_artifacts

Fixes # (issue)

LIN-592

Type of change

How Has This Been Tested?

New tests were added.

yoonspark commented 1 year ago

Confirmed that this works for the following example (which previously resulted in an error):

import lineapy

art = {}
b0 = 0
art["b0"] = lineapy.save(b0, "b0")
a = b0+1
art["a"] = lineapy.save(a, "a")

lineapy.to_pipeline(
    artifacts=["a", "b0"],
    dependencies={"a":{"b0"}},
    reuse_pre_computed_artifacts=["b0"],
)