Solution would be to create a custom ingress, that uses the job_id in the URL. The user knows the job_id, so it's easy to access the Spark UI.
An extra reason to deploy a custom Ingress is to be able to use TLS with a signed certificate. The current situation only allows self-signed certs.
The current situation is as follows:
A batch job is started, with a random ID. This random ID is then used in the URL that's exposed via the Ingress. An example is: https://a-595d70270def4079986fde9e626424cd.spark-jobs-staging.batch.stag.warsaw.openeo.dataspace.copernicus.eu This ID is however unknown to the user, so it's impossible to use the Spark UI.
Solution would be to create a custom ingress, that uses the
job_idin the URL. The user knows the job_id, so it's easy to access the Spark UI. An extra reason to deploy a custom Ingress is to be able to use TLS with a signed certificate. The current situation only allows self-signed certs.