search

huawei-noah / streamDM

Stream Data Mining Library for Spark Streaming
http://streamdm.noahlab.com.hk/
Apache License 2.0
492 stars 147 forks source link

StreamDM-81: Update to Spark 2.2.0 #82

Closed hmgomes closed 6 years ago

hmgomes commented 6 years ago

Summary of the changes

The changes in this pull request were made to update streamDM to the current Spark version, i.e., Spark 2.2.0.

Most classes in the framework were touched by this request, mostly on the import section. Unused imports were removed as well. Other small changes were performed as well, since all classes were revisited, such as opportunities to use val instead of var.

sbt.learn was also updated to include the new versions for the existing dependencies.

Tests

Just a sanity check to verify if the code is working with the latest version. The test uses the electNormNew.arff, which is inside \data directory, so there is no need to include other dataset.

  1. Execute EvaluatePrequential using electNormNew.arff

Run the command:

./spark.sh "200 EvaluatePrequential -l (trees.HoeffdingTree -l 0 -t 0.05 -g 200 -o) -s (FileReader -f ../data/electNormNew.arff -k 1000 -d 10 -i 10000) -e (BasicClassificationEvaluator -c -m) -h" 1> result_ELEC.txt 2> log_ELEC.log

Expected output: 10 rows of statistics in results_ELEC.txt

JianfengQian commented 6 years ago

Sounds great.