Passing in model_kwargs

[derpyplops]

Hey there, I wanted some help running eq-bench. specifically i wanted to test different sampler settings (min_p). however i noticed in the configs that there's no way to specify model_kwargs / gen_kwargs, am I right? I'm not familiar with ooba, so maybe they're passed in that way? In that case I'd have to make a fork to pass in the configs. If it's useful maybe i can make a PR. Is my understanding correct?

[sam-paech]

Hi there! I'm not supporting min_p directly (this is by design so that the benchmark runs with repeatable params). But you can specify sampler config with ooba like this:

# lib/run_query.py

def run_ooba_query(...):
...
data = {
"mode": "instruct",        
"messages": messages,
"instruction_template": prompt_format,
"max_tokens": completion_tokens,
"temperature": temp,
"user_bio": "",
"min_p": 0.1
}

You may also wish to hardcode temp here if you are using min_p. EQ-Bench uses temp 0.01 by default, and the creative writing test uses temp 0.7