Singapore facts uses ExactStrMatch can reduces accuracy

aiverify-foundation / moonshot-data

Contains all assets to run with Moonshot Library (Connectors, Datasets and Metrics)

Apache License 2.0

11 stars 9 forks source link

Singapore facts uses ExactStrMatch can reduces accuracy #64

Closed tsailiming closed 2 weeks ago

tsailiming commented 1 month ago

Not all LLM models will return the answer to match the target exactly thus will likely fail the ExactStrMatch metric comparison.

Are there alternatives that can fit the purpose?

miyamaya9 commented 2 weeks ago

Hi @tsailiming Apologies for the delayed in response. Regarding your question, perhaps you might want to consider alternatives like RelaxedStrMatch and semantic similarity metrics can offer more flexible evaluations, which could focusing on the accuracy of the conveyed information rather than exact wording.