search

meta-llama / llama

Inference code for Llama models
Other
55.37k stars 9.44k forks source link

Reproduce ToxiGen eval for llama2? #723

Open sdujq opened 1 year ago

sdujq commented 1 year ago

I tried to reproduce the evaluation on ToxiGen Dataset, but failed (both Llama-2-7b-hf and Llama-2-13b-hf)

shots: 6-shots dataset: https://github.com/microsoft/SafeNLP/blob/main/data/toxiGen.json generation params: top_p 0.9, temperature = 0.1, max_new_tokens 32 toxi cls model: tomh/toxigen_roberta (and count LABEL_1)

result in paper: 21.25% but i got: 39.95%

anything i can do? thanks

TJKlein commented 1 year ago

I also tried all kinds of combinations of data (6-shot, 1-shot), format, and generation (greedy, top_p), but cannot reproduce.

TJKlein commented 1 year ago

Evaluating in 1-shot manner in combination with top_p = 1.0 and temperature = 1.0 I get something like 20.5% toxicity for a llama-7b model.

Jellyfish042 commented 12 months ago

My attempt to reproduce has also failed. May I ask how your prompt was specifically constructed?

huu4ontocord commented 9 months ago

Hello - any progress on this. It's really important that we can reproduce these safety eval so we can trust this really awesome model ;)

tangbinh commented 9 months ago

For ToxiGen, we generated text in a zero-shot setting.