search

AbsaOSS / spline

Data Lineage Tracking And Visualization Solution
https://absaoss.github.io/spline/
Apache License 2.0
603 stars 155 forks source link

Report: missing lineage for data source #735

Open DaimonPl opened 4 years ago

DaimonPl commented 4 years ago

Just an idea, in our current setup spline is not enabled everywhere (currently on purpose, initial phase).

It would be helpful if there was report in UI showing which datasources do not have their lineage - sorted by number of usages

So if i have 3 jobs which are using datasource "users" and i don't have lineage for "users" itself, report could show that this datasource with that URI is used by 3 jobs but lineage is not available

wajda commented 4 years ago

Good idea We'll think about it in a scope of a new UI Meantime you can use the following AQL query to achieve that:

FOR ds IN dataSource
    LET lineage_cnt = LENGTH(FOR a IN affects FILTER a._to == ds._id RETURN ds)
    LET impacts_cnt = LENGTH(FOR d IN depends FILTER d._to == ds._id RETURN ds)
    FILTER lineage_cnt == 0
    RETURN [impacts_cnt, ds.uri]
wajda commented 3 years ago

related to #804