search

microsoft / promptbench

A unified evaluation framework for large language models
http://aka.ms/promptbench
MIT License
2.45k stars 182 forks source link

About Semantic Attacks Against Vicuna #8

Closed tianshuocong closed 1 year ago

tianshuocong commented 1 year ago

Hi!

Thanks for your nice work!

One small question is that when I visited your demo site, and I choose "Vicuna" + "MNLI" + “Semantic” + “zero-shot task”, the site return nothing. Therefore, I am confused and could you please replenish these adversarial prompts? Many thanks!

Best wishes

pb-issue
Immortalise commented 1 year ago

Hi,

Thank you for bringing this error to our attention! We will update the attack results shortly.

tianshuocong commented 1 year ago

Hi!

Thank you very much!

I also noticed that when I selected model as Vicuna, and selected dataset squad_v2，UN Multi, and math, there are also not adversarial prompts.

Immortalise commented 1 year ago

Yes, Vicuna performs much worse on these datasets, we choose not to evaluate Vicuna on these datasets.