The changes in this pull request were made to update streamDM to the current Spark version, i.e., Spark 2.3.2. It shall also works with Spark 2.3.1 and Spark 2.3.
Class src/main/scala/org/apache/spark/streamdm/core/NullInstance.scala was updated to address a small concern due to the Scala 2.11 version.
sbt.learn was updated to include the new versions for the existing dependencies.
If the log is outputted to stdout (by default it is) the result output file from the test command will contain both the results and the log. One way to avoid that is by explicitly setting the log4j.properties file in Spark home (just remove “template” from log4j.properties.template in /conf in the Spark diretory).
spark.sh was updated to point to the new Scala 2.11 jar file
It was tested using Scala 2.11.8
This is related to #102
Tests
Just a sanity check to verify if the code is working with the latest version.
The test uses the electNormNew.arff, which is inside \data directory, so there is no need to download a dataset to run the experiment.
Execute EvaluatePrequential using electNormNew.arff
Summary of the changes
The changes in this pull request were made to update streamDM to the current Spark version, i.e.,
Spark 2.3.2. It shall also works with Spark 2.3.1 and Spark 2.3.sbt.learnwas updated to include the new versions for the existing dependencies.spark.shwas updated to point to the new Scala 2.11 jar fileTests
Just a sanity check to verify if the code is working with the latest version. The test uses the electNormNew.arff, which is inside
\datadirectory, so there is no need to download a dataset to run the experiment.Run the command:
Expected output: 10 rows of statistics in results_ELEC.txt