Results of classifier examination

[MatthiasBaur]

Some information about the process: To find the best possible classifier we took the dataset, chose a reasonable size (10000 instances) and added all the features we had. Then we ran a lot of different configurations and tried to find improvement by omitting, filtering, adding, discretizing and oversampling instances. On the classifier side we combined, schemed and varied variables. For a more detailed report look into the ClassfierData branch. There are the tools for reproducing results.

Results.xlsx

In this file you can see a overview over the selection without weather features. The first tables holds most results, the second holds some attribute evaluations and the fourth experiments (bundled up runs). The SMOTE table's result proved useless when tested against a separate testing set. The fifth table can be used to play around with the confidence factor on a example classifier's results.

This data is NOT representing every configuration run. Not at all. The times stated are almost always in reference to training 1 fold, so it excludes the testing time and the other 9 folds.

Today we made the final reduction. The results will be (hopefully) confirmed tonight and posted tomorrow.

Greetings, The Predict Team

[goldbergtatyana]

Hi @MatthiasBaur, please provide us here an overview of:

1. the best performing classifier and its parameters
2. the features you found to be contributing to best results
3. the performance within a ten-fold cross-validation on the 10k data set using the model from 1. and features from 2.
4. the performance using the same 10 test sets (a 1k each), same model and same features, but larger training set (49k), as we discussed on Friday. The reason for this test is to check if a larger taining set leads to better prediction results. To see this effect, the test sets of 3. and 4. have to stay equal.
5. a prediction probabiliy threshold, below which you say a pokemon is not being predicted and we need the performance at this threshold as well, of course :)

Thank you!

[MatthiasBaur]

1. the best performing classifier and its parameters

Our best classifier is a voting scheme using IBk and a BayesNet. See http://weka.sourceforge.net/doc.dev/weka/classifiers/meta/Vote.html http://weka.sourceforge.net/doc.dev/weka/classifiers/lazy/IBk.html http://weka.sourceforge.net/doc.dev/weka/classifiers/bayes/BayesNet.html

The non-default parameters in Weka are: Vote (-R PROD) , IBk (-K 100)

1. the features you found to be contributing to best results

latitude, longitude, appearedMinute, appearedDayOfWeek, closeToWater, urban, midurban, rural, gymIn100m, gymIn1000m, pokestopIn250m, pokestopIn2500m, cooc_13, cooc_16, cooc_19, cooc_96, cooc_129, class

We also combined the rarest pokemon in one single class. @semioniy worked out this list: 1,4,7,10,11,13,14,15,16,17,18,19,20,21,22,23,25,27,29,30,32,33,35,37,39,41,42,43,44,46,47,48,49,50,52,54,56,58,60,61,63,66,69,72,73,74,75,77,79,81,84,85,86,90,92,95,96,97,98,100,102,104,109,111,114,116,118,119,120,123,124,127,128,129,133,138,143,147 These are the top 78 pokemon. The rest will be combined.

1. the performance within a ten-fold cross-validation on the 10k data set using the model from 1. and features from 2.

results_10k_1.txt

-> ~ 21 %

1. "..."

I ran the tests we discussed. The 50 k dataset generally performed equal to the 10 k set (by ~ 0.1 % worse on average). I have the test runs available should you need them.

1. "..."

This really depends on how many icons can be displayed efficiently on the smarphone screen. @PokemonGoers/pokemap-2 Can you test this? E.g. if you have a zoom level of 250x250m, how many icons would look good? Or would you like to have all the data, including the prediction probability and then vary the number of pokemons displayed based on the zoom level?

[goldbergtatyana]

The 50 k dataset generally performed equal to the 10 k set (by ~ 0.1 % worse on average). Sweet!

I have the test runs available should you need them. Yes, please upload them here on github

[goldbergtatyana]

since we trained a new classifier, we can close this issue.