search

cerndb / spark-dashboard

Spark-Dashboard is a solution for monitoring Apache Spark jobs. This repository provides the tooling and configuration for deploying an Apache Spark Performance Dashboard using containers technology.
Apache License 2.0
111 stars 23 forks source link

Inconsistency in Spark Dashboard #2

Closed mittal-yash closed 1 year ago

mittal-yash commented 1 year ago

Hi Luca,

I am using Spark Dashboard to monitor spark jobs that are using spark 2.2.0. We are running jobs in yarn. In cluster mode inconsistency is observed. For some jobs metrics is showing in dashboard and for some jobs metrics is not coming. Can you please guide.

LucaCanali commented 1 year ago

Hi, I am glad to hear you are using the Spark Dashboard. However, I am afraid Spark 2.2.0 is quite an old version of Spark and several metrics used in the dashboard have been introduced in Spark 2.3.0 and 2.4.0. Best, Luca

mittal-yash commented 1 year ago

Hi Luca,

Can you please suggest how to enable logs in spark dashboard.

LucaCanali commented 1 year ago

I am afraid that the are no specific logs for the dashboard itself, but it relies on logging from the components (Spark, Influx, Grafana).

mittal-yash commented 1 year ago

Thank you for the info