Create Reporting layer - Githubissues

databrickslabs / overwatch

Capture deep metrics on one or all assets within a Databricks workspace

Other

230 stars 64 forks source link

Create Reporting layer #1155

Open gueniai opened 9 months ago

gueniai commented 9 months ago

Let's start building our reporting (or platinum) layer. We should start with the main dimensions (job, cluster, notebok, etc), plus a simplified and aggregated view of clsf and jrcp, to make it easy to get daily aggregates. Something we'll have to take into account will be in identifying changes to rows we've already processed, since there could be slight changes in gold due to late arriving events.

Stories

[ ] closes #1276
[ ] closes #1289
[ ] closes #1277
[ ] closes #1294

neilbest-db commented 5 months ago

Is there a branch and/or PR for this? Just curious. Could not find anything.

neilbest-db commented 5 months ago

I guess the folder here(/Workspace/Users/sourav.banerjee@databricks.com/1. OVERWATCH/Platinum Layer) in overwatch-field must be the work in progress.