Fixed fake rate normalization

[LucioDerin]

Summary

This pull request introduces the following changes

• The fake count is now normalized over all samples predicted to be from class $c$, independently of their true origin (before it was normalized over samples from class $c$ correctly predicted, which didn't guarantee a normalization to 1).

Conformity

• [x] Changelog entry
• [x] Documentation

[samvanstroud]

I'm struggling a bit here mainly because I don't know what scp.sparse.coo_matrix is doing.

I think I probably should have asked for the efficiency and purity rather than efficiency and fake rate here though. Is there any chance we can use the standard functions from sklearn here instead?

• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html
• https://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html

I think this is probably a good idea in case people want to easily calculate other metrics.

[dkobylianskii]

I'm struggling a bit here mainly because I don't know what scp.sparse.coo_matrix is doing.

Our confusion_matrix.py file is just a simplification of the sklearn.metrics.confusion_matrix function. We did this to reduce the number of dependencies because otherwise, we need the whole library for just one function.

What do you think about it?

[samvanstroud]

@dkobylianskii true, although I can imagine we might find other useful functions in sklearn, and there is not really a massive cost to adding it as a dep.

But since we have things working in our implementation I'm happy to keep things the way they are! @LucioDerin if you can just make sure we are computing the purity of the classification that would be perfect.

Thanks both!

[LucioDerin]

But since we have things working in our implementation I'm happy to keep things the way they are! @LucioDerin if you can just make sure we are computing the purity of the classification that would be perfect.

@samvanstroud I am happy to replicate sklearn's functions to compute the precision and recall scores, so that we have scores with clear definitions. Otherwise I can implement a function to compute the purity. Let me know what you think is best :)

[samvanstroud]

Precision and purity are the same, I'm happy to go with whichever approach you think is simplest, thanks @LucioDerin

[LucioDerin]

Precision and purity are the same, I'm happy to go with whichever approach you think is simplest, thanks @LucioDerin

I will try to implement those metrics in puma :) Maybe it's better to close this PR: I'll open another one as soon as the code is ready. Let me know what you think about that! @samvanstroud

[samvanstroud]

sounds good thanks @LucioDerin !