For spark multi-version support, change setAll to set as setAll doesn't exist in Spark 2.4.
Temporarily set SPARK_HOME to the directory corresponding to the Spark version, as spark submit -- version will first find the SPARK_HOME of the current environment, rather than the SPARK_HOME corresponding to the current directory.
Does this pull request potentially affect one of the following parts
Dependencies (does it add or upgrade a dependency): (yes / no)
What changes were proposed in this pull request
Brief change log
Verifying this change
Custom Code
toSpark Jar
.setAll
toset
assetAll
doesn't exist in Spark 2.4.SPARK_HOME
to the directory corresponding to the Spark version, asspark submit -- version
will first find theSPARK_HOME
of the current environment, rather than theSPARK_HOME
corresponding to the current directory.Does this pull request potentially affect one of the following parts