search

databrickslabs / overwatch

Capture deep metrics on one or all assets within a Databricks workspace
Other
221 stars 60 forks source link

Add descriptive Job Group ID and `NamedTransformation`s to Spark UI #1223

Open neilbest-db opened 1 month ago

neilbest-db commented 1 month ago

This makes the Spark UI more developer-friendly when analyzing Overwatch runs.

Job group IDs have the form :

Any use of .transform( df => df) may be replaced with .transformWithDescription( nt) after instantiating a val nt = NamedTransformation( df => df) as its argument.

This commit contains one such application of the new extension method. (See val jobRunsAppendClusterName in WorkflowsTransforms.scala.)

Some logic in GoldTransforms falls through to elements of the special job-run-action form of Job Group IDs emitted by the platform but the impact is minimal relative to the benefit to Overwatch development and troubleshooting. Even so this form of Job Group ID is still present in initial Spark events before OW ETL modules begin to execute.

sonarcloud[bot] commented 1 month ago

Quality Gate Passed Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

neilbest-db commented 3 weeks ago

@souravbaner-da, @gueniai, is there anything I can do to help get this merged in?

sonarcloud[bot] commented 3 days ago

Quality Gate Passed Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

neilbest-db commented 3 days ago

closes #1226

neilbest-db commented 3 days ago

this should be merged into 0820_release before #1228