LineaLabs / lineapy

Move fast from data science prototype to pipeline. Capture, analyze, and transform messy notebooks into data pipelines with just two lines of code.
https://lineapy.org
Apache License 2.0
664 stars 58 forks source link

Fix session-level TaskDefinition #860

Closed andycui97 closed 1 year ago

andycui97 commented 1 year ago

Description

Session level task definitions were generated incorrectly, the returned artifacts dictionary was not unpacked before saving individual artifacts to their respective locations causing jobs to fail.

This PR fixes this and adds airflow tests for this case.

Type of change

How Has This Been Tested?

Two new airflow tests. Note: This makes the airflow test run for quite a bit longer, but still should be less than the total time of the normal test suite.