At the moment we are rethrowing the exception in cases there were issues in getting Application ID from the YarnBackend. We should instead return optional empty and just log the exception in order to allow calling code to decide what to do if application Id could not be fetched.
Also we need to add RuntimeException which is thrown sometimes by Hadoop/Spark libs:
java.lang.RuntimeException: Exception in thread "main" java.io.IOException: Could not get block locations. Source file "/user/xxxxx/.sparkStaging/application_1698161930067_305157/xxxxx" - Aborting...block==null
at org.apache.spark.launcher.OutputRedirector.redirect(OutputRedirector.java:67)
at java.base/java.lang.Thread.run(Unknown Source)
...
At the moment we are rethrowing the exception in cases there were issues in getting Application ID from the YarnBackend. We should instead return optional empty and just log the exception in order to allow calling code to decide what to do if application Id could not be fetched.
Also we need to add RuntimeException which is thrown sometimes by Hadoop/Spark libs: