Meeting 11/3/2015

[Moorkopsoesje]

Done:

• @Lucivius and @Guardianofnature have tested the classifier (SGD) with different parameters and the performance is bad. The log loss went from infinity to 4.0, so best whale pauper. @robbertvdg also tested Luc's and Steven's code for the SGD on the RBM and the log loss was also at best 4.1 (when tested on 1/50 of data, 4.7 when tested on all data), so we are going to try another clsfr (SVM is widely used so first this one). Although this is only tested on one setting for the parameters yet.
• Since the benchmark is around 4, the SGD seems to have random results. This might be because this isn't a good classifier for this data, but there probably is an error somewhere. So we could also look really carefully at the code.
• @robbertvdg fixed the RBM. At first all the weights had about the same value. Together with @StevenReitsma he looked at it, now instead of using the exponent function just the weights x #visible units are used + normalized.
• @Lucivius and @Guardianofnature didn't normalize the k-means yet because they thought this was already done in the preprocessing.
• Normalization is a bit of an issue, since normalizing the patches would mean that the data would become too large. So we want to normalize the images before extracting the patches. This would also increase the memory usage a lot, but less than normalizing the patches.
• Normalizing the patches would also probably be a bit useless, since the average patch would probably be white because of all the edges of the images which are white. Normalizing the images before extracting patches makes more sense.
• A way to combine the CNN and method of Coates might be a weighted average, the weights could possibly be determined using the entropy. Since just taking the highest probabilites doesn't say much if high probabilities aren't that informative, so this might be an easy way to combine the methods.

To-do:

• [x] @Lucivius and @Guardianofnature are going to implement the SVM classifier before this Saturday so it's ready for the Hackathon.
• [x] @robbertvdg and @Moorkopsoesje are going to run the RBM with different parameters
• [x] @Rahazan is going to look at an efficient way for normalization with help from @robbertvdg
• [x] Look into calculating the entropy of the output.
• [x] Look into more classifiers we could use.
• [x] Carefully look at the code for the SGD to see if we can find mistakes which cause the bad performance.

[StevenReitsma]

Some pointers:

• For the SVM, use scikit-learn's SVC class with probabilities=True
• Combining the models can be a linear average, weighted average or the Coates features can be directly inputted in the CNN (I'd have to write a custom layer for this, but it's possible). This would remove the need of a classifier completely, but for testing we do need the classifier. The linear average is the easiest, and if the Coates performs well enough (<1.0 loss) it might be the best option.
• I'm not sure how you would determine the weights using entropy.

[Moorkopsoesje]

Well, the entropy is a measure of uncertainty so if the probabilities for the classes are all about the same, then it's not very informative. Just picking the highest probabilities is not really a good way to choose which method you want to use to assign a class to an image, you need the classification method that is the most certain of the most probable class relative to the other probabilities for classes. So calculating the entropy for the output of the CNN and the entropy for the output of Coates method, could then help decide for which prediction you should go, right?

[StevenReitsma]

We could definitely take a look at it, but I'm not sure whether it will work since certainty and correctness are not necessarily equal.