LineaLabs / lineapy

Move fast from data science prototype to pipeline. Capture, analyze, and transform messy notebooks into data pipelines with just two lines of code.

https://lineapy.org

Apache License 2.0

662 stars 58 forks source link

Implement pipeline writers for artifact collection (includes PR-737) #738

Closed yoonspark closed 2 years ago

yoonspark commented 2 years ago

Description

Implement pipeline writers for SCRIPT (LIN-477) and AIRFLOW (LIN-484) frameworks. These writers use new graph refactoring implemented by ArtifactCollection for pipeline file generation; and they are meant to eventually replace existing plugins.

NOTE: This PR "absorbs" #737.

Type of change

[x] New feature (non-breaking change which adds functionality)

How Has This Been Tested?

New unit tests were added to validate if pipeline files are properly generated.

andycui97 commented 2 years ago

General comment, how were the "expected" files generated, and how did we validate that they are correct?

yoonspark commented 2 years ago

@andycui97 The "expected" files were generated by running the test scenarios over local Jupyter notebook sessions; after manually inspecting/testing they look good and runnable, these files were copied to the test folder as "expected" results.