AjayBathini-MSFT
commented
1 year ago @YassinElf Thanks for bringing this to our attention. I'm going to assign this to the document author so they can take a look at it accordingly.
Open royal-gala opened 1 year ago
AjayBathini-MSFT
commented
1 year ago @YassinElf Thanks for bringing this to our attention. I'm going to assign this to the document author so they can take a look at it accordingly.
As of right now, in order to monitor Azure Databricks clusters, there is this page containing info from Microsoft: https://learn.microsoft.com/en-us/azure/architecture/databricks-monitoring/application-logs
It is outlined how to get cluster logs from Azure Databricks clusters to a Log Analytics workspace, which is my goal. However, the issue I am experiencing is that we will be running clusters with runtime versions 11.0 (Spark 3.3.0) and above, which is not supported by MS' library Spark Monitoring (https://github.com/mspnp/spark-monitoring).
Is there any recommendation as to how to export databricks clusters logs to a log analytics workspace ?
Many thanks!
Document Details
⚠ Do not edit this section. It is required for learn.microsoft.com ➟ GitHub issue linking.