huawei-noah / streamDM

Stream Data Mining Library for Spark Streaming
http://streamdm.noahlab.com.hk/
Apache License 2.0
492 stars 147 forks source link

Update stream dm job - batchInterval is not hardcoded anymore #71

Closed nhnminh closed 7 years ago

nhnminh commented 7 years ago

This pull request has one change in streamDMJob.scala:

Change:

Test: To verify this, you could run the following test:

  1. Test with batchInterval:

./spark.sh "200 EvaluatePrequential -l (trees.HoeffdingTree -l 0 -t 0.05 -g 200 -o) -s (FileReader -f ../data/electNormNew.arff -k 4000 -d 10)" 1> result.res 2> log.log

This sets batchInterval to be 200 ms

  1. Test with negative batchInterval:

./spark.sh "-300 EvaluatePrequential -l (trees.HoeffdingTree -l 0 -t 0.05 -g 200 -o) -s (FileReader -f ../data/electNormNew.arff -k 4000 -d 10)" 1> result.res 2> log.log

This sets batchInterval to be default value = 1000 ms

  1. Test with no user-defined value: ./spark.sh "EvaluatePrequential -l (trees.HoeffdingTree -l 0 -t 0.05 -g 200 -o) -s (FileReader -f ../data/electNormNew.arff -k 4000 -d 10)" 1> result.res 2> log.log

This sets batchInterval to be default value = 1000 ms