Spark job template builder

[oliver-tarrant-tessella]

Given a complete set of appropriate arguments, this service produces the script body that is submitted to the k8s spark-operator.

Acceptance criteria

• a POST request can be submitted to the submit-job handler with the following arguments: [job_name, language (python, scala etc), path_to_main_application_file, executors, driver_cores, driver_memory, executor_cores, executor_memory, arguments]
  • depending on the language, extra arguments may be required (e.g. mainClass for scala)
• Returns a dictionary object with the arguments with the sparkapplication definition

Notes

No validation is performed at this stage:

• If required arguments are given then an exception should be thrown
• Resources for drivers and executors are unlimited at this stage

[robert-clegg-tessella]

System Tests Submit Spark Pi Job Returns Ok Response and Can Get Logs Of Submitted Spark Job confirm that a spark job can be submitted via the web app, meaning that the web app is successfully building manifest files.