Updating eval for lm_eval 0.4 and 0.3

[HDCharles]

Stack from ghstack (oldest at bottom):

• 83

• 97

• -> #91

Summary: 0.4 broke BC, this fixes regardless of version

Test Plan: (on both versions and without lm_eval installed)

python quantize.py --mode int8

(on both versions)

python eval.py --tasks wikitext

wikitext: {'word_perplexity,none': 12.212490471702079, 'word_perplexity_stderr,none': 'N/A', 'byte_perplexity,none': 1.59675331009031, 'byte_perplexity_stderr,none': 'N/A', 'bits_per_byte,none': 0.6751414412399839, 'bits_per_byte_stderr,none': 'N/A', 'alias': 'wikitext'}

For model checkpoints/meta-llama/Llama-2-7b-chat-hf/model.pth wikitext: {'word_perplexity': 12.212490471702079, 'byte_perplexity': 1.59675331009031, 'bits_per_byte': 0.6751414412399839}

Reviewers:

Subscribers:

Tasks:

Tags: