search

cerndb / spark-dashboard

Spark-Dashboard is a solution for monitoring Apache Spark jobs. This repository provides the tooling and configuration for deploying an Apache Spark Performance Dashboard using containers technology.
Apache License 2.0
111 stars 23 forks source link

does this integrates with databricks #1

Closed hardiktalati closed 1 year ago

hardiktalati commented 2 years ago

Hey Luca,

It offers more like overwatch from databricks - i use stagemetrics with databricks and localhost. I would like to do the same with spark-dashboard, deploy docker on container instance in azure and somehow integrate with databricks... is it possible or i am on a wrong track.

LucaCanali commented 2 years ago

I dont' have the possibility to test this at present, however, it should work. Spark dashboard uses the Apache Spark metrics system with the GraphiteSink, then an instance of InfluxDB 1.x is used in my implementation to capture those metrics. As long as Spark (driver and executors) can communicate with InfluxDB (typically on port 2003) that should work.