The version of Spark, configuration and associated JAR files needs to be refreshed.
Spark version
We should look for a more recent version in line with supported Dataproc versions.
JAR files
We should find a more recent combination of JAR files but also cater for not including the JAR files at all, it is better to supply package addresses via offload.env and not host JAR files.
Perhaps we need to support both.
Configuration
We can assume a GCP target in the short term so could include those configuration items in the default fiules.
Or perhaps cater for it by having multiple make targets.
The version of Spark, configuration and associated JAR files needs to be refreshed.
Spark version We should look for a more recent version in line with supported Dataproc versions.
JAR files We should find a more recent combination of JAR files but also cater for not including the JAR files at all, it is better to supply package addresses via offload.env and not host JAR files. Perhaps we need to support both.
Configuration We can assume a GCP target in the short term so could include those configuration items in the default fiules. Or perhaps cater for it by having multiple make targets.