Open sdujq opened 1 year ago
I also tried all kinds of combinations of data (6-shot, 1-shot), format, and generation (greedy, top_p), but cannot reproduce.
Evaluating in 1-shot manner in combination with top_p = 1.0 and temperature = 1.0 I get something like 20.5% toxicity for a llama-7b model.
My attempt to reproduce has also failed. May I ask how your prompt was specifically constructed?
Hello - any progress on this. It's really important that we can reproduce these safety eval so we can trust this really awesome model ;)
For ToxiGen
, we generated text in a zero-shot setting.
I tried to reproduce the evaluation on ToxiGen Dataset, but failed (both Llama-2-7b-hf and Llama-2-13b-hf)
shots: 6-shots dataset: https://github.com/microsoft/SafeNLP/blob/main/data/toxiGen.json generation params: top_p 0.9, temperature = 0.1, max_new_tokens 32 toxi cls model: tomh/toxigen_roberta (and count LABEL_1)
result in paper: 21.25% but i got: 39.95%
anything i can do? thanks