search

IDEA-FinAI / ToG

This is the official github repo of Think-on-Graph. If you are interested in our work or willing to join our research team in Shenzhen, please feel free to contact us by email (xuchengjin@idea.edu.cn)
238 stars 26 forks source link

Some confusion about eval #22

Open willer-lu opened 3 months ago

willer-lu commented 3 months ago

For example, if the ground truth is 'University Yale', but the output of llm is 'Yale University', or '01.01' vs '1st January'.
Such instance should be considered right or wrong when caculating EM?

GasolSun36 commented 3 months ago

Hi, It's definitely wrong, so this is a shortcoming of EM. However, you can use turbo to determine whether the calculation is correct or incorrect given the ground-truth and the generated result.