search

exacaster / lighter

REST API for Apache Spark on K8S or YARN
MIT License
91 stars 21 forks source link

Return Optional.empty() instead of rethrowing exception and catch RuntimeException when submiting #824

Closed edvaone closed 9 months ago

edvaone commented 9 months ago

At the moment we are rethrowing the exception in cases there were issues in getting Application ID from the YarnBackend. We should instead return optional empty and just log the exception in order to allow calling code to decide what to do if application Id could not be fetched.

Also we need to add RuntimeException which is thrown sometimes by Hadoop/Spark libs:

java.lang.RuntimeException: Exception in thread "main" java.io.IOException: Could not get block locations. Source file "/user/xxxxx/.sparkStaging/application_1698161930067_305157/xxxxx" - Aborting...block==null
        at org.apache.spark.launcher.OutputRedirector.redirect(OutputRedirector.java:67)
        at java.base/java.lang.Thread.run(Unknown Source)
...