search

hltfbk / Excitement-Open-Platform

Excitement Open Platform for Recognizing Textual Entailments
http://hltfbk.github.io/Excitement-Open-Platform/
86 stars 74 forks source link

Derive the score of entailment #546

Open elrusa opened 8 years ago

elrusa commented 8 years ago

Hi,

I wonder how it's possible to convert a confidence of either ENTAILMENT or NONENTAILMENT from the result file into just a score of ENTAILMENT, such that

1 - the system is sure that the pair is ENTAILMENT. 0 - the system is sure that the pair is NONENTAILMENT.

As result, I want to use these scores as feature values.

rzanoli commented 8 years ago

The confidence is a value in the range is [0,1], and 1 means full confidence. Note that the confidence is about the decision reported in this decision object, and it is not the probability of entailment.

From my poin of view if you want to use these scores as feature values for further classification you might think to have an attribute or feature saying if there is entailment or not (ENTAILMENT, NONENTAILMENT) and a second attribute for the feature weighting (the confidence provided by the classifier). Another possibility would be to use the classifier output as it is (ENTAILMENT, NONENTAILMENT) without considering any confidence score, or you could instead think to apply a filter to increse the precision; e.g., you might decide that if the confidence provided by the classifier is higher then 80% then the ENTAILMENT feature is just the label provided by the classifier(ENTAILMENT, NONENTAILMENT) ; contrary if the confidence is below 80% then the feature label is UNKNOWN.