search

huawei-noah / streamDM

Stream Data Mining Library for Spark Streaming
http://streamdm.noahlab.com.hk/
Apache License 2.0
492 stars 147 forks source link

StreamDM-100 #101

Open ioanna-ki opened 6 years ago

ioanna-ki commented 6 years ago

Bug report #100 - part of the solution

1. remove duplicates of the output coreset

2. fixed the break statement

still needs some fixes in StreamKM

Summary of the changes

BucketManager's getCoreset returned a lot of duplicate Examples in the resulted coreset. Almost over 50% of the examples was duplicates. With these changes this percentage is around 10% but not zero.

in splitCoresetTreeLeaf sometimes the one child had all the points and the other was empty. Now it rechooses a center and splits the points again.

in splitCoresetTree changed the way of choosing the which node to split. Similar to moa's treeCoreset in BucketManager the break statement in getCoreset does not work properly. Now there is a flag that stops the process when a full bucket is found.

Tests

need to print the resulted coreset in StreamKM be sure getCoreset is invoked at least once

CLAassistant commented 3 years ago

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.