Add the option to submit spark jobs with arguments. These will be passed to the spark job in the order as supplied. Arguments must be provided as an array but the type of the values in the array are unrestricted.

Stories covered:

45

To test

Ensure copy of app is running with the latest code

Run the built in Scala job Group by test using the following body in your request:

{
"name": "group-by-test3",
"language": "Scala",
"path_to_main_app_file": "local:///opt/spark/examples/jars/spark-examples_2.11-2.4.0.jar",
"main_class": "org.apache.spark.examples.GroupByTest",
"arguments": ["11", "2052", "750", "3" ],
"executors": "5",
"executor_cores": "1",
"driver_cores": "0.2",
"driver_memory": "512m",
"executor_memory": "1048m"
}

Vary arguments and see how it affects the logs
Run system tests and unit tests

ukaea / piezo

Feature/submit job accepts argument input as array/#45 #62

Stories covered:

45

To test