Closed mianqiu00 closed 3 months ago
Mrpc
zero-shot
python inference.py -m Llama-2-7b-hf -d mrpc -shots 0 --ranking_type prob
Our results: 31.62.
python inference.py -m Meta-Llama-3-8B -d mrpc -shots 0 --ranking_type prob
Our results: 43.38.
one-shot
python inference.py -m Llama-2-7b-hf -d mrpc -shots 1 --ranking_type prob
Our results: 46.32.
python inference.py -m Meta-Llama-3-8B -d mrpc -shots 1 --ranking_type prob
Our results: 69.61.
32-shot
python inference.py -m Llama-2-7b-hf -d mrpc -shots 32 --ranking_type prob
Our results: 42.65.
python inference.py -m Meta-Llama-3-8B -d mrpc -shots 32 --ranking_type prob
Our results: 69.85.
prob is implemented in ranking type instead of get_pll, but identical prediction can not be avoid thoroughly, llama3 performs much better in this task
Mrpc
zero-shot
Our results: 31.62.
Our results: 43.38.
one-shot
Our results: 46.32.
Our results: 69.61.
32-shot
Our results: 42.65.
Our results: 69.85.
prob is implemented in ranking type instead of get_pll, but identical prediction can not be avoid thoroughly, llama3 performs much better in this task