search

tomkerkhove / promitor

Bringing Azure Monitor metrics where you need them.
https://promitor.io
MIT License
249 stars 91 forks source link

Provide scraper for Azure DataBricks #2117

Open gburton1 opened 2 years ago

gburton1 commented 2 years ago

Proposal

@tomkerkhove do you have any thoughts on whether there could be a generalized, viable way to monitor Azure DataBricks with Promitor? If there was a clean pathway, we could contribute. Azure officially offers this approach: https://docs.microsoft.com/en-us/azure/architecture/databricks-monitoring/

We are exploring alternatives as well, such as exposing Prometheus metrics directly on the Spark nodes.

Component

Scraper

Contact Details

No response

tomkerkhove commented 2 years ago

Unfortunately it does not look like Databricks is providing metrics in Azure Monitor - https://docs.microsoft.com/en-us/azure/azure-monitor/essentials/metrics-supported

locmai commented 2 years ago

Yes, Databricks service is using the new way to export the metrics via Azure Log Analytics. This might help - https://github.com/tomkerkhove/promitor/issues/1076

Related discussion: https://github.com/tomkerkhove/promitor/discussions/1975