Adds support for kubeflow pipelines. Setting framework= KUBEFLOW in to_pipeline now produces a dag file that can be run through the kubeflow pipeline sdk.
Also adds task serialization that can be parametrized so that the save location can be specific through the task function signature. Kubeflow's get_rendered_task_definitions uses this by adding these locations as function parameters to user_input_variables.
Fixes LIN-724
Type of change
Please delete options that are not relevant.
[X] New feature (non-breaking change which adds functionality)
Description
Adds support for kubeflow pipelines. Setting
framework= KUBEFLOWinto_pipelinenow produces a dag file that can be run through the kubeflow pipeline sdk.Also adds task serialization that can be parametrized so that the save location can be specific through the task function signature. Kubeflow's
get_rendered_task_definitionsuses this by adding these locations as function parameters touser_input_variables.Fixes LIN-724
Type of change
Please delete options that are not relevant.
How Has This Been Tested?
Using housing example on local kubeflow setup.