LineaLabs / lineapy

Move fast from data science prototype to pipeline. Capture, analyze, and transform messy notebooks into data pipelines with just two lines of code.

https://lineapy.org

Apache License 2.0

661 stars 58 forks source link

LIN-738: Add new Ray workflows API as the default #866

Closed andycui97 closed 1 year ago

andycui97 commented 1 year ago

Description

Adds new RAY workflow as the default for RAY pipelines. This change modifies the behavior described in https://github.com/LineaLabs/lineapy/pull/862

Ray now supports use_workflows, setting to True (which is also default) uses the new Ray workflows API
Adds ability to configure the runtime_env and storage options in ray.init()
Updated tests since some workflows are not representable in new Ray workflows API

Fixes LIN-738

Type of change

Please delete options that are not relevant.

[X] New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Ray tests updated. Run with housing example using both "use_workflows" as True and False to ensure both new and old DAGs run.