Add support for decoding binary classifications

[dlebech]

This PR adds support for decoding predictions from networks where the final layer has exactly one output dimension and two output classes (assumed to be between 0 or 1).

This is useful for binary classification networks. I found that for binary classification, a single-output prediction layer works better than having two outputs, which is why I think it made sense to add a special case for this, but I understand if you feel this is too specific. I thought I would open a PR just in case :slightly_smiling_face:

Thanks for this awesome tool! :beers: I'm using it over here.

[jakebian]

Thanks for the PR!

I'm not sure what problem you're trying to fix here: what goes wrong if you just run quiver with 1 output class?

[dlebech]

@jakebian the problem is that I have two output classes that just map to a single 0/1 output.

So my classes could be ['dog', 'cat'] but the output is just a single number between 0 (dog) and 1 (cat). This means the shape of the output is (X, 1) instead of (X, 2). For example, a prediction of e.g. 0.7 would be "cat" and e.g. 0.4 would be dog. I guess in the "normal" case, the output for these examples would be [0.3, 0.7] and [0.6, 0.4].

As I mentioned, this might be a specific use case, but I found the binary prediction to perform better with a single output instead of using two outputs, and I thought why not see if it made sense for a PR as well :slightly_smiling_face:

[jakebian]

Ah I see what you're saying, prior to this fix it would have always said "dog" even when the output is 0.01.

I'm reluctant to merge this as is because it is a little silly to ALWAYS ask the user to "name" the negative case for binary classification problems. For example when a user runs quiver on their "is this a picture of a pizza" classifier, in addition to the label "isPizza", after this fix they MUST also add a class that says "isNotPizza".

There's a simple fix for this:

Simple solution: make this feature optional.

However, there's a simple generalization of this feature that could be useful in more cases.

Simple generalization

In any classification problem, the number 1 - (sum of all the outputs) provides a measure of "how likely it is that the correct class lies outside of all the classes provided". Hence it could be helpful to give the user an option to always label this number.

For example, if you have trained a classifier with (genuinely) 2 outputs, cat, dog, and you feed it a picture of a desk lamp, most likely the model will produce a pair of very small numbers, say (0.1, 0.2). It would be maybe helpful to label the number 1 - 0.1 - 0.2 = 0.7 as neither. Furthermore if you only ever expect inputs to be either dogs, cats or lamps, you can label this case lamp.

Concretely, this would be a new OPTIONAL field in the configuration that looks something like

{"complementLabel": "lamp"}

which defaults to None.

Should be pretty straight-forward to tweak your code to implement this, and it would make a pretty cool feature!

[dlebech]

@jakebian thank you for your thoughtful response. You made me re-think my design decision of having two classes for one output. I'll close this PR for now, and give it some more thought. Hopefully, I'll have time to make something that conforms better with your suggestion of having a complement label rather than explicitly saying that "not X" is an actual class.