prompts for CS-SNLI - Githubissues

DCGM / lm-evaluation-harness

A framework for few-shot evaluation of language models.

https://www.eleuther.ai

MIT License

0 stars 2 forks source link

Closed popelucha closed 2 months ago

popelucha commented 3 months ago

not sure about the metrics (got acc=0)
not sure about the correct use of the gold label (got the message "Label index was not in within range of available choices,Sample")

MFajcik commented 2 months ago

Seems to be running fine! Thank you @popelucha!

MFajcik commented 2 months ago

@popelucha, while not a breaking change, I noticed that https://huggingface.co/datasets/CZLC/cs_snli/tree/main contains test.jsonl.jsonl . I suspect that is a typo :).