Include the deployment of services and service monitors (to gather metrics from spark applications) with the deployment of the web app. If service already exists and is up to date then it remains unchanged else it is updated or deployed. This opens up a passage for all metrics produced in the spark job pods to be visible in the Prometheus dashboard
Testing
In prometheus it is now possible to see new jobs appear under Targets/default/spark-application-monitor/0 as they are launched. The metrics currently being exposed align with those visable by running kubectl port-forward {jobs driver or executor pods} 8090 and then navigating in the browser to localhost:8090.
However, there are still missing metrics but this will be addressed in #23 and #71 as this is due to configuring the spark image and operator.
Include the deployment of services and service monitors (to gather metrics from spark applications) with the deployment of the web app. If service already exists and is up to date then it remains unchanged else it is updated or deployed. This opens up a passage for all metrics produced in the spark job pods to be visible in the Prometheus dashboard
Testing
In prometheus it is now possible to see new jobs appear under
Targets/default/spark-application-monitor/0as they are launched. The metrics currently being exposed align with those visable by runningkubectl port-forward {jobs driver or executor pods} 8090and then navigating in the browser tolocalhost:8090. However, there are still missing metrics but this will be addressed in #23 and #71 as this is due to configuring the spark image and operator.