search

mspnp / spark-monitoring

Monitoring Azure Databricks jobs
MIT License
213 stars 178 forks source link

spark-monitoring.sh throws error with DBR 11 #202

Closed gfree76 closed 1 year ago

gfree76 commented 1 year ago

When I try to use DBR 11 with our existing shell-monitoring.sh, I get errors that say 

BEGIN: Updating /databricks/spark/dbconf/log4j/executor/log4j.properties with Log Analytics appender
bash: line 18: StreamingQueryListenerSampleJob: command not found
bash: line 18: FS_CONF_COMPAT: command not found
sed: can't read /databricks/spark/dbconf/log4j/executor/log4j.properties: No such file or directory

If I restart the cluster without the init-script, I see that there is no log4j.properties, although there is a log4j.xml

Is there a version of the shell-monitoring.sh that will work with the log4j.xml?

vinnypaluch commented 1 year ago

maybe because of this?

⚠️ | This library supports Azure Databricks 10.x (Spark 3.2.x) and earlier (see Supported configurations). Azure Databricks 11.0 includes breaking changes to the logging systems that the spark-monitoring library integrates with. The work required to update the spark-monitoring library to support Azure Databricks 11.0 (Spark 3.3.0) and newer is not currently planned

matthiaspfenninger commented 1 year ago

(See also #187 which has been opened in July 2022 – it does not look like this is a priority.)

gfree76 commented 1 year ago

(See also #187 which has been opened in July 2022 – it does not look like this is a priority.)

Thanks Matthias - I'll close this issue and go pile on in #187