Lineage service

[mcdoker18]

Implement Lineage service or use existed

• [ ] Implement lineage service from the scratch OR integrate existed one into ODAHU (UI integration for example) see https://github.com/google/ml-metadata for example

Enhance ODAHU API

• [ ] ODAHU API should log stages into Lineage service using adapter (Lineage service should be pluggable)
• [ ] ODAHU API should accept ExperimentName to ensure grouping of different runs into "folders"
• [ ] ODAHU API should accept PipelineRunId to ensure grouping of different stages into single pipeline run

[karbyshevds]

We need to provide separate API that will allow users to pass metadata from different ODAHU workflow steps to it using unique identifier. The main goal is to be able to track all workflow activity by ID on all steps.

[alinaignatiuk]

Non-trivial logging structured for particular pipeline which allows tracking steps status across the ML pipeline (prediction, production pipelines). The main goal is debugging to understand what goes wrong The next is to regulation of the processes inside (what is going on and whether processes are compliant) 3r party solutions primary (MLflow logging, sagemaker) goal is to track the ML specific steps status kubeflow pipeline need research for https://github.com/google/ml-metadata, however we should into account ODAHU does not have own pipeline service orchestration service? LP