The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.
Is your feature request related to a problem? Please describe.
Runtimes by model are great for seeing individually inefficient models but another thing that is often needed to optimise the lineage between the models that ultimately affects total uptime of a dbt run. You can have many efficient models individually but if they all run concurrently you will have a very long dbt task.
Describe the solution you'd like
To identify these problems areas with Lineage I would like to see a lineage graph laid out like a Gant chart with model start and end times so you can see which models are key choke points in the lineage (perhaps as they are depended on by many models etc.)
Extras:
A view of lineage depth - i.e. how many models deep a specified node is from source / exposure - this gives an idea of project complexity
A view on lineage effeciency - some sort of metric of how well the nodes in a project are designed to run in parallel.
Describe alternatives you've considered
We developed this tool internally but it was too much to maintain with the changes to artefact structure over time. The below images are a very rough alpha version of how we imagined this might be presented.
Additional context
Would you be willing to contribute this feature?
Definitely if I can with some guidance from the team.
Maayan-s
commented
1 year ago
Is your feature request related to a problem? Please describe. Runtimes by model are great for seeing individually inefficient models but another thing that is often needed to optimise the lineage between the models that ultimately affects total uptime of a dbt run. You can have many efficient models individually but if they all run concurrently you will have a very long dbt task.
Describe the solution you'd like To identify these problems areas with Lineage I would like to see a lineage graph laid out like a Gant chart with model start and end times so you can see which models are key choke points in the lineage (perhaps as they are depended on by many models etc.)
Extras:
Describe alternatives you've considered We developed this tool internally but it was too much to maintain with the changes to artefact structure over time. The below images are a very rough alpha version of how we imagined this might be presented.
Additional context
Would you be willing to contribute this feature? Definitely if I can with some guidance from the team.
ELE-442