search

databrickslabs / overwatch

Capture deep metrics on one or all assets within a Databricks workspace
Other
221 stars 60 forks source link

[UPGRADE] - Upgrade spark compiler to 3.3.2 #868

Open GeekSheikh opened 1 year ago

GeekSheikh commented 1 year ago

[Should be implemented with #878 #985 ]

Current compiler is on 3.1.2 --> should be upgraded to 3.3.2

This is causing several tests to fail -- tests need to be repaired as well.

ConcurrentHashSet is no longer available either so the globalTableLock will need to switch to mutable.LinkedHashSet in SparkSessionWrapper

private[overwatch] val globalTableLock = new mutable.LinkedHashSet[String]

neilbest-db commented 2 weeks ago

Quoting this comment in #985:

I have linked branch compile_for_dbr_11_3 here as an intermediate step to DBR 12.2. Both are LTS releases, having end-of-support dates of 2025-10-19 and 2026-03-01 respectively. Respective Spark versions are 3.3.0 and 3.3.2 so aiming for 12.2 here makes more sense than jumping straight to 13.3 LTS (Spark 3.4.1, EOS 2026-08-01).

Same thinking and linked feature branch apply here.