search

JonasGeiping / cramming

Cramming the training of a (BERT-type) language model into limited compute.
MIT License
1.29k stars 100 forks source link

Cola dataset evaluation #21

Closed TahaBinhuraib closed 1 year ago

TahaBinhuraib commented 1 year ago
image

Hello, when I use the evaluation script I get this for the cola dataset but all looks well for the other datasets. Also, when I look at the logs, I can see the Matthews correlation.

image
JonasGeiping commented 1 year ago

ugh, thanks for letting me know! What version of evaluate is this using?

TahaBinhuraib commented 1 year ago

I'll take a look ASAP, but would I be able to use the metric reported in the logs?

JonasGeiping commented 1 year ago

yeah, the metric from the logs is also fine. The .yaml file is just mistyped - somehow a python object is written into the file, instead of a float value. If you figure out what the object class is , then you could probably even reload the number correctly from the yaml file.

JonasGeiping commented 1 year ago

Also fixed in https://github.com/JonasGeiping/cramming/releases/tag/Torch2.1