Update the monitoring pages in the wiki to give an overview of what is available to the users for optimising their spark applications and how to use/find the relative information for their jobs using the spark ui and prometheus.
Due to Grafana being removed in favour of the spark ui we are not going to provide detailed instructions for Grafana. However, steps should be outlined for how Grafana could potentially be integrated in the future.
Acceptance criteria
Instructions on the Monitoring wiki page are clear
Update the monitoring pages in the wiki to give an overview of what is available to the users for optimising their spark applications and how to use/find the relative information for their jobs using the spark ui and prometheus.
Due to Grafana being removed in favour of the spark ui we are not going to provide detailed instructions for Grafana. However, steps should be outlined for how Grafana could potentially be integrated in the future.
Acceptance criteria
Instructions on the Monitoring wiki page are clear