Clarifying GLUE Benchmark Accuracy: Validation or Test Set?

jiaweizzhao / GaLore

GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection

Apache License 2.0

1.24k stars 131 forks source link

Clarifying GLUE Benchmark Accuracy: Validation or Test Set? #13

Closed monk1337 closed 3 months ago

monk1337 commented 3 months ago

Hello, I enjoyed reading your excellent paper. I have one question: is the accuracy mentioned for the GLUE benchmarks based on the validation set or the test set? The paper does not specify this detail. Thank you for your clarification.

Kyriection commented 3 months ago

Hi, Thanks for your interest in our work. The results of GLUE benchmarks are reported on the validation set, following the implementation from https://github.com/huggingface/transformers/blob/main/examples/pytorch/text-classification/run_glue.py.