How do you calculate ACC?

iKala / ievals

Official github repo for TMMLU+, Large scale traditional chinese massive multitask language understanding

MIT License

43 stars 2 forks source link

How do you calculate ACC? #1

Open GitYCC opened 9 months ago

GitYCC commented 9 months ago

How do you calculate those ACC in below figure?

How to calculate STEM, Social Science, Humanities, Other accuracy? What are the subjects in STEM, Social Science, Humanities, Other ?

Can you share subject.tsv file mentioned in https://github.com/ikala-corp/ievals/blob/main/ievals/settings.py#L85 ?

ray2ikala commented 9 months ago

Sorry @GitYCC I have added the config and subject names for it.

The accuracy was first calculated according to the sub category in subjects.tsv and average within the major subjects. Finally all four major subjects are averaged to get the final average score