search

brmson / yodaqa

A Question Answering system built on top of the Apache UIMA framework.
http://ailao.eu/yodaqa
Other
619 stars 205 forks source link

Why correct answer is not getting the highest confidence score? #55

Closed satyagadepalli closed 8 years ago

satyagadepalli commented 8 years ago

We have found that answers with 2nd highest confidence level is actually the correct than the 1st highest answer.

For example, If we ask "text : "Who is the Inventor of Computer?"" we got the following response ( for space limitation i am showing the top 2).

{"snippetIDs":[15051,15089],"confidence":0.802490288802869,"text":"René Sommer","ID":26775},{"snippetIDs":[15093],"confidence":0.6304183601686735,"text":"computer Charles Babbage","ID":26791}

Are we missing something here?

Thanks

pasky commented 8 years ago

Unfortunately the question answering system is not perfect and makes mistakes (sometimes or often, depending on the kinds of questions asked). The pipeline can be modified to suit better a specific domain or kind of questions... But it'll always be a "fuzzy" system that answers some questions wrong.

We have found that answers with 2nd highest confidence level is actually the correct than the 1st highest answer.

For example, If we ask "text : "Who is the Inventor of Computer?"" we got the following response ( for space limitation i am showing the top 2).

{"snippetIDs":[15051,15089],"confidence":0.802490288802869,"text":"René Sommer","ID":26775},{"snippetIDs":[15093],"confidence":0.6304183601686735,"text":"computer Charles Babbage","ID":26791}

Are we missing something here?

Thanks

You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/brmson/yodaqa/issues/55