search

databrickslabs / overwatch

Capture deep metrics on one or all assets within a Databricks workspace
Other
226 stars 64 forks source link

[BUG] OW fails when ETL DB name has a hyphen #858

Open GeekSheikh opened 1 year ago

GeekSheikh commented 1 year ago

Must be completed after #868

spark.catalog.databaseExists(dbName) does not work if dbName contains hyphens. This can be resolved by adding back ticks inside the quotes. Example in screenshot and text is below.

This will fix the issue

spark.catalog.databaseExists(s"`${dbName}`")

Be sure to check for use of this function throughout the code base and apply this fix to all usages.

https://github.com/databrickslabs/overwatch/blob/5337cadd510f596b03914ea48066db0f5df21641/src/main/scala/com/databricks/labs/overwatch/pipeline/Initializer.scala#L415

image

souravbaner-da commented 1 year ago

Currently this issue will be on hold. As the suggested solution depend on spark version. Specifically from spark version 3.3.0 the solution is working fine and for all other version the solution is failing. Only solution for this is to check spark version during the check of database exist and build our solution depend on the spark version. May be look into it in future release.

For now in overwatch public doc we will mention that database name in config should not contain any special character. Corresponding issue for the documentation is

874