search

Waikato / moa

MOA is an open source framework for Big Data stream mining. It includes a collection of machine learning algorithms (classification, regression, clustering, outlier detection, concept drift detection and recommender systems) and tools for evaluation.
http://moa.cms.waikato.ac.nz/
GNU General Public License v3.0
609 stars 353 forks source link

The prequential accuracy drops to 50% steadily after ADWIN detects a drift. #192

Closed michaelchiucw closed 1 year ago

michaelchiucw commented 4 years ago

Hi there,

I have found an issue about the ADWIN change detector. The ADWIN change detector in the latest MOA release (MOA 19.05.0) will always drops the prequential accuracy to 50% steadily. This issue is regardless the base learner.

You should be able to recreate this issue by simply create a concept drift stream and use the DriftDetectionMethodClassifier with using ADWIN as change detector. You can pick any base learner as you like, say, the most basic one: Naive bayes. For the evaluator, I used FadingFactor with default alpha 0.999. With these settings, you will then see the performance chart will drop to 50% quickly after the drift detection. The performance will then keep at 50% extremely steady.

The ADWIN change detector was fine in version 18.06, as I have been using this version of MOA in my research.

michaelchiucw commented 4 years ago

Screenshot 2019-11-21 at 23 55 46 Screenshot 2019-11-21 at 23 54 42

michaelchiucw commented 4 years ago

The first image is MOA-18.06 The second image is MOA-19.05 (latest version) Stream: 100,000 instances 0-50,000: Agrawal f1 balanced class 50,001-100,000: Agrawal f10 balanced class The rest of the setting I didn't mention here are in MOA default.

michaelchiucw commented 1 year ago

I have recently tried this with the latest version of MOA (2023.04.0). This issue has been resolved.