Exposes the spark ui to users so that they can monitor the progress of their spark applications to helm with optimisation. The Ui is exposed via dynamically generating a proxy for each job that communicates with the spark ui service created by the spark operator. A service is also created to allow communications with the proxy and a ingress rule is generated to provide a unique url route for each ui. The url to the ui is returned to the user when they submit their job. If any of the required components fails to create successfully the url is returned as Unavailable.
Note the spark ui is only accessible for the duration that the spark job is running. It can also take a bit of time for the proxy, ui and services to initialise and so the spark ui may not be available for the first few seconds.
Tidy up of the Ui
Upon deletion of a job all components are deleted also deleted. If any part is failed to be deleted then it is logged. If this occurs they will need manually deleting with kubectl. To make this easier all components are labled with release=piezo so that they can all be deleted together without touching other kubernetes components.
Documenting decisions and monitoring
Due to ingress constraints a decision has been made to expose the spark ui instead of Grafana. The details have been outlined in the ADR. As part of #91 we will outline how to use the spark-ui together with prometheus for monitoring spark applications and outline possible solutions for integrating Grafana again in the future.
Expose the spark ui
Exposes the spark ui to users so that they can monitor the progress of their spark applications to helm with optimisation. The Ui is exposed via dynamically generating a proxy for each job that communicates with the spark ui service created by the spark operator. A service is also created to allow communications with the proxy and a ingress rule is generated to provide a unique url route for each ui. The url to the ui is returned to the user when they submit their job. If any of the required components fails to create successfully the url is returned as
Unavailable.Note the spark ui is only accessible for the duration that the spark job is running. It can also take a bit of time for the proxy, ui and services to initialise and so the spark ui may not be available for the first few seconds.
Tidy up of the Ui
Upon deletion of a job all components are deleted also deleted. If any part is failed to be deleted then it is logged. If this occurs they will need manually deleting with
kubectl. To make this easier all components are labled withrelease=piezoso that they can all be deleted together without touching other kubernetes components.Documenting decisions and monitoring
Due to ingress constraints a decision has been made to expose the spark ui instead of Grafana. The details have been outlined in the ADR. As part of #91 we will outline how to use the spark-ui together with prometheus for monitoring spark applications and outline possible solutions for integrating Grafana again in the future.
To test
Stories covered
99