One of the core features we can deliver with KubeETL is metadata + lineage collection. This issue is to discuss options + proposals surrounding metadata collection.
Currently we are evaluating OpenLineage and OpenMetadata as potential targets for Metadata collection.
We have several options to gather metadata:
Sync DataSet + Workflow state with the metadata store.
Create Crawlers to crawl e.g. database, file storage and store the metadata in the metadata store.
Metadata can also be used to sync back to KubeETL:
One of the core features we can deliver with KubeETL is metadata + lineage collection. This issue is to discuss options + proposals surrounding metadata collection.
Currently we are evaluating OpenLineage and OpenMetadata as potential targets for Metadata collection.
We have several options to gather metadata:
Metadata can also be used to sync back to KubeETL: