search

likenneth / honest_llama

Inference-Time Intervention: Eliciting Truthful Answers from a Language Model
MIT License
461 stars 36 forks source link

Query on result of Vicuna and Alpaca #17

Closed jongjyh closed 1 year ago

jongjyh commented 1 year ago

Hi!

I'm trying to compare my method to ITI with Alpaca and Vicuna, But I failed to reproduce the result identical to paper, Could you please share the results and hyperparemeters on this repo to reproduce the result on Alpaca and Vicuna since it's a little bit different since the first commit.

FYI, I successfully reproduced LLaMA!

Thank you very much!

Bests,

jongjyh commented 1 year ago

Do these result append the few-shot prompting? I guess problem maybe comes from here?

Alpaca + ITI 65.1 66.6 31.9 2.92 0.61 Vicuna + ITI 74.0 88.6 38.9 3.36 1.41

likenneth commented 1 year ago

Hi, there.

If you've reproduced llama results, it should be very easy for vicuna and alpaca. The only thing you need to change is the model_name parameter given to get_activation.py and validate_2fold.py.

KL