LineaLabs / lineapy

Move fast from data science prototype to pipeline. Capture, analyze, and transform messy notebooks into data pipelines with just two lines of code.
https://lineapy.org
Apache License 2.0
664 stars 58 forks source link

LIN-705: Return taskgraph for pipeline integrations to have correct data dependencies #854

Closed andycui97 closed 1 year ago

andycui97 commented 1 year ago

Description

Fixes LIN-705

Type of change

How Has This Been Tested?

Added new diamond test which snapshotted should cover dependencies specified between artifacts in different sessions to point to the upstream root deps of the target artifact. Specifically a >> a >> linear_first_for_artifact_linear_second_and_downstream even though the dependency is defined as {"linear_third": {"a"}}. For more details refer to discussion in this thread.

Update snapshots and manually verify a few that they are correct for changes.