search

elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.71k stars 24.67k forks source link

Data Lineage/dependency management #73801

Open sherry-ger opened 3 years ago

sherry-ger commented 3 years ago

Data Lineage for transform/rollup jobs so one can understand the dependencies. It would be useful for both primary and downstream consumers to understand the relationship. As a primary index owner I update the schema, I understand this may impact X transforms that are mapped to Y index mapped to Z index owner. Knowing the data lineage will prevent users accidentally breaking transform or rollup jobs.

elasticmachine commented 3 years ago

Pinging @elastic/es-analytics-geo (Team:Analytics)

elasticmachine commented 3 years ago

Pinging @elastic/ml-core (Team:ML)

wchaparro commented 2 years ago

removing the analytics/rollups label as this isn't something we can support in rollups. I think this issue for ML transforms- captures the request.