[EPIC] - Micro-Pipeline - Faster Delivery for Key Components (P1 - 720)

databrickslabs / overwatch

Capture deep metrics on one or all assets within a Databricks workspace

Other

230 stars 64 forks source link

[EPIC] - Micro-Pipeline - Faster Delivery for Key Components (P1 - 720) #425

Open GeekSheikh opened 2 years ago

GeekSheikh commented 2 years ago

Many customers want certain datasets at a much higher frequency than is currently supported through Overwatch.

Some examples include Cluster Events, Job Runs, DBSQL Queries, etc. This data can only be accumulated and delivered through the APIs today but allowing for trickle load jobs to run much more frequently than the Overwatch pipeline can facilitate higher-resolution, low-cost solution to these requirements.

GeekSheikh commented 2 years ago

Sprint 1

jobRun
clusterEvents

Sprint 2

endpointEvents
queryHistory
instancePools
etc..

GeekSheikh commented 2 years ago

Reference

Pre-requisites PRs #432 #381 #415 (all of these should merge into 6.1.2)

Issues #431 and #433 in PR #432