Make multiclass prediction score available in Python

[mdredze]

A prediction can be one of several types: https://github.com/JohnLangford/vowpal_wabbit/blob/master/vowpalwabbit/example.h#L37

When we have a multi-class prediction, it populates the field multiclass. If --probabilities is provided, it populates probs of prob.

It is often helpful, even without using probabilities, to have access to the score associated with a multiclass prediction. For example, if you want to set a prediction threshold you need the score.

Since multi-class predictions only store the label, we don't have access to the score.

I have a proposed change. Replace uint32_t multiclass with MULTICLASS::label_t multiclass. MULTICLASS::label_t has two fields: a label (the prediction) and a weight, which we can use to store the score.

While this requires changing a dozen lines of code, its impact is minimal. Everywhere that currently accessed multiclass will instead access multiclass.label. The prediction method in oaa is modified to store the score. Finally, the Python interface is modified to make the prediction score available.

If this sounds good, I can generate a PR.

[jconwell]

Exposing all scores for all labels in multiclass prediction would be really useful as well. Essentially exposing the same output as --raw_predictions in the python API

[mdredze]

I considered doing that. A few options. 1) A list of floats, with the ith float being the ith labels score. This would require every method that accessed this list to find the max. 2) An ordered list of MULTICLASS::label_t. If you wanted the prediction, you'd access the 0th entry. Otherwise, you can iterate through the list to find the value you want. 3) Store both the ordered list as in #1 AND the predicted label. Makes it efficient to get the max prediction, and any score.

Thoughts?

[eisber]

I like (3), but can we parameterize to avoid additional overhead if one doesn’t need one or the other? I’m using a factory pattern in C# for the predicitons to trigger what I need…

From: mdredze [mailto:notifications@github.com] Sent: Wednesday, January 27, 2016 4:50 PM To: JohnLangford/vowpal_wabbit vowpal_wabbit@noreply.github.com Subject: Re: [vowpal_wabbit] Make multiclass prediction score available in Python (#918)

I considered doing that. A few options. 1) A list of floats, with the ith float being the ith labels score. This would require every method that accessed this list to find the max. 2) An ordered list of MULTICLASS::label_t. If you wanted the prediction, you'd access the 0th entry. Otherwise, you can iterate through the list to find the value you want. 3) Store both the ordered list as in #1https://github.com/JohnLangford/vowpal_wabbit/pull/1 AND the predicted label. Makes it efficient to get the max prediction, and any score.

Thoughts?

— Reply to this email directly or view it on GitHubhttps://github.com/JohnLangford/vowpal_wabbit/issues/918#issuecomment-175874079.

[mdredze]

There is currently a flag --probabilities that stores probability scores. We could add a similar flag that stores the raw scores.

On Wednesday, January 27, 2016, Markus Cozowicz notifications@github.com wrote:

I like (3), but can we parameterize to avoid additional overhead if one doesn’t need one or the other? I’m using a factory pattern in C# for the predicitons to trigger what I need…

From: mdredze [mailto:notifications@github.com javascript:_e(%7B%7D,'cvml','notifications@github.com');] Sent: Wednesday, January 27, 2016 4:50 PM To: JohnLangford/vowpal_wabbit <vowpal_wabbit@noreply.github.com javascript:_e(%7B%7D,'cvml','vowpal_wabbit@noreply.github.com');> Subject: Re: [vowpal_wabbit] Make multiclass prediction score available in Python (#918)

I considered doing that. A few options. 1) A list of floats, with the ith float being the ith labels score. This would require every method that accessed this list to find the max. 2) An ordered list of MULTICLASS::label_t. If you wanted the prediction, you'd access the 0th entry. Otherwise, you can iterate through the list to find the value you want. 3) Store both the ordered list as in #1< https://github.com/JohnLangford/vowpal_wabbit/pull/1> AND the predicted label. Makes it efficient to get the max prediction, and any score.

Thoughts?

— Reply to this email directly or view it on GitHub< https://github.com/JohnLangford/vowpal_wabbit/issues/918#issuecomment-175874079>.

— Reply to this email directly or view it on GitHub https://github.com/JohnLangford/vowpal_wabbit/issues/918#issuecomment-175886616 .

[jconwell]

I like (3) as well. And would love the raw scores

[JohnLangford]

We discussed these variants when --probabilities was being made. I'd suggest just tweaking things so that '--scores' provides arbitrary scores: use the same datastructure as --probabilities, but no logistic link function.

There is little computational value in providing the scores and the max (option (3)), because the computation of the scores deeply amortizes a max operation. So, if we wanted to provide a max (or min), I'd think that should be done by function rather than by RAM.

-John

On 01/27/2016 05:20 PM, John Conwell wrote:

I like (3) as well. And would love the raw scores

— Reply to this email directly or view it on GitHub https://github.com/JohnLangford/vowpal_wabbit/issues/918#issuecomment-175890719.

[jconwell]

I'm down with a '--scores' flag that populates *probs. It would then be easy to add an accessor function to pylibvw.cc to return the scores or probs, depending on the flag.

[JohnLangford]

There have been several changes to the code to support more advanced prediction types of one sort or another so this is relatively easy to do now. This: https://github.com/JohnLangford/vowpal_wabbit/blob/master/vowpalwabbit/example.h#L42 is probably the right/best datastructure to store raw scores in for the purpose of reading out from the library interface.

[danqing]

Wondering if there's any progress/plan on this :) I think having the raw scores is going to be immensely helpful

[JohnLangford]

I've modified oaa.cc to allow you to pass --scores which stores all the scores. Hence, the underlying plumbing is now setup, and this may just work in some interfaces.

[JohnLangford]

I've modified oaa.cc to allow you to pass --scores which stores all the scores. Hence, the underlying plumbing is now setup, and this may just work. I'm not positive about the python interface. @gramhagen does the python interface multiplex on the prediction type stored here: https://github.com/JohnLangford/vowpal_wabbit/blob/master/vowpalwabbit/learner.h#L114 ? (This is relatively recent, but it makes interfaces much cleaner/safer.)

[gramhagen]

the python interface multiplexes on a provided label type, which can be accessed from the label parser (vw->p->lp). It shouldn't be too hard to switch this to the learner's prediction type and provide access to raw scores through the python interface. #1139 already starts to provide multiple class probabilities.

[gramhagen]

This is now supported through the python wrapper with #1139, using scores will give you a list of probabilities.