Materialized views for lineage events graph

phixMe commented 2 months ago

Problem

We wish to enable counts for lineage events via a new graph on our main application page!

Solution

Creates two materialized views that get updated via Crons managed through Dropwizard at respective intervals.

We are supporting daily and hourly rollups for now.

Note: This PR does not included current timeframe rollups.

Checklist

[x] You've signed-off your work
[ ] Your changes are accompanied by tests (if relevant)
[ ] Your change contains a small diff and is self-contained
[ ] You've updated any relevant documentation (if relevant)
[ ] You've included a one-line summary of your change for the CHANGELOG.md (Depending on the change, this may not be necessary).
[ ] You've versioned your .sql database schema migration according to Flyway's naming convention (if relevant)
[ ] You've included a header in any source code files (if relevant)

netlify[bot] commented 2 months ago

Deploy Preview for peppy-sprite-186812 canceled.

Name	Link
Latest commit	ed5353376764bda959bf3e7f5bf80dc508d69b30
Latest deploy log	https://app.netlify.com/sites/peppy-sprite-186812/deploys/66e0c197e192d700088fa818

codecov[bot] commented 2 months ago

Codecov Report

Attention: Patch coverage is 53.12500% with 30 lines in your changes missing coverage. Please review.

Project coverage is 83.00%. Comparing base (915aff3) to head (bc6d5f5). Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
...java/marquez/jobs/MaterializeViewRefresherJob.java	57.57%	14 Missing :warning:
...ava/marquez/db/mappers/LineageMetricRowMapper.java	0.00%	9 Missing :warning:
api/src/main/java/marquez/api/StatsResource.java	44.44%	5 Missing :warning:
...pi/src/main/java/marquez/service/StatsService.java	60.00%	2 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #2891 +/- ## ============================================ - Coverage 83.28% 83.00% -0.29% - Complexity 1477 1486 +9 ============================================ Files 259 264 +5 Lines 6785 6849 +64 Branches 313 315 +2 ============================================ + Hits 5651 5685 +34 - Misses 977 1007 +30 Partials 157 157 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

dolfinus commented 2 months ago

What is represented by this plot on the image? What do these colors mean, why there are 3 on them? What are x and y axis are showing?

phixMe commented 2 months ago

Hey, this isn't quite available in our interfaces yet, but we're working on the totality of the page. It's designed to keep track of the types of lineage events coming into the system... There are various tooltips built into the UI to describe what is going on as well as button hovers.

https://github.com/user-attachments/assets/0061ae05-e621-4412-8f43-ca5a9629e9b3

MarquezProject / marquez