Is this benchmark being maintained?

THU-KEG / KoLA

[ICLR24] The open-source repo of THU-KEG's KoLA benchmark.

https://arxiv.org/abs/2306.09296

50 stars 0 forks source link

Is this benchmark being maintained? #14

Open ZhangzihanGit opened 7 months ago

ZhangzihanGit commented 7 months ago

Hi,

It seems like the benchmark is not being actively maintained -- the latest session is October 2023. Could you please let me know if you are planning to maintain it as said in your paper?

In addition, I understand that,

KoLA does not allow the local evaluation to prevent test set leakage and unfairly overfitting datasets (cited from the paper)

however, do you have plans to open-source the evaluation data from previous seasons?

Thanks.

xurh20 commented 6 months ago

Yes it should be actively maintained. Not really, as if there's a new model our plan is to test it through all previous seasons data - in which case open-source them will lead to data-leackage.

xurh20 commented 5 months ago

Hi, just fyi that the newest s3 test result is updated on the website. You can have a check if you are interested, thx!