AreaUnderRoc - "Problem Evaluating Classifier: Null"

[BrianKCL]

Hi,

I am using AutoWeka and selecting "areaUnderRoc" as a metric. I've run it for over an hour, for testing, and before the hour passes, it gives me an error window of "Problem Evaluating Classifier: Null". I am using Ubuntu and this is the output in terminal:

[Thread-21] INFO autoweka.TrajectoryMerger - Experiment /tmp/autoweka1641756550$ [Thread-21] INFO weka.classifiers.meta.AutoWEKAClassifier - Thread 0, best conf$ [Thread-21] INFO autoweka.TrajectoryMerger - Experiment /tmp/autoweka8688787055$ [Thread-21] INFO weka.classifiers.meta.AutoWEKAClassifier - Thread 1, best conf$ [Thread-21] INFO autoweka.TrajectoryMerger - Experiment /tmp/autoweka5721229934$ [Thread-21] INFO weka.classifiers.meta.AutoWEKAClassifier - Thread 2, best conf$ [Thread-21] INFO autoweka.TrajectoryMerger - Experiment /tmp/autoweka5497810617$ [Thread-21] INFO weka.classifiers.meta.AutoWEKAClassifier - Thread 3, best conf$ [Thread-21] INFO weka.classifiers.meta.AutoWEKAClassifier - classifier: null, a$ java.lang.NullPointerException

I use cross validation of 10 folds. In the editor, I have the following: batchSize: 100 debug: False doNotCheckCapabilities: False memLimit: 1024 metric: areaUnderROC nBestConfigs: 1 numDecimalPlaces: 2 parallelRuns: 4 seed: 123 timeLimit: 60

I've also run it for 16 hours and it gives me the same error for the test data sets. I've used "weather.numeric.arff".

Thank You!

[BrianKCL]

The output in terminal got cut. This is the complete output.

[Thread-21] INFO autoweka.TrajectoryMerger - Experiment /tmp/autoweka1641756550631321875/Auto-WEKA [Thread-21] INFO weka.classifiers.meta.AutoWEKAClassifier - Thread 0, best configuration estimate 1.0 [Thread-21] INFO autoweka.TrajectoryMerger - Experiment /tmp/autoweka8688787055697354729/Auto-WEKA [Thread-21] INFO weka.classifiers.meta.AutoWEKAClassifier - Thread 1, best configuration estimate 1.0 [Thread-21] INFO autoweka.TrajectoryMerger - Experiment /tmp/autoweka5721229934524335582/Auto-WEKA [Thread-21] INFO weka.classifiers.meta.AutoWEKAClassifier - Thread 2, best configuration estimate 1.0 [Thread-21] INFO autoweka.TrajectoryMerger - Experiment /tmp/autoweka549781061736005314/Auto-WEKA [Thread-21] INFO weka.classifiers.meta.AutoWEKAClassifier - Thread 3, best configuration estimate 1.0 [Thread-21] INFO weka.classifiers.meta.AutoWEKAClassifier - classifier: null, arguments: [], attribute search: null, attribute search arguments: [], attribute evaluation: null, attribute evaluation arguments: [] java.lang.NullPointerException

[BrianKCL]

I've found the error. I was using a data set that had a relative small amount of instances for one cluster. 10 for one and 190 for the other. By doing a more balanced partition 75/125, it did not produce this error.

[larskotthoff]

Great to see that it's working now!