Open kortemik opened 10 months ago
https://blog.cloudera.com/enforcing-application-lifetime-slas-yarn/ describes a method for setting lifetime programmatically using following:
ApplicationSubmissionContext#setApplicationTimeouts(Map<ApplicationTimeoutType, Long> applicationTimeouts)
https://hadoop.apache.org/docs/r3.0.2/api/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.html#setApplicationTimeouts-java.util.Map-
however spark-submit does not seem to support it: https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L263
https://issues.apache.org/jira/browse/SPARK-46238
kortemik
commented
10 months ago kortemik
commented
10 months ago
https://blog.cloudera.com/enforcing-application-lifetime-slas-yarn/ describes a method for setting lifetime programmatically using following:
https://hadoop.apache.org/docs/r3.0.2/api/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.html#setApplicationTimeouts-java.util.Map-
however spark-submit does not seem to support it: https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L263