search

yuzu-ai / japanese-llm-ranking

Apache License 2.0
47 stars 12 forks source link

`do_sample` is unused in `gen_model_answer.py` #16

Closed toslunar closed 10 months ago

toslunar commented 10 months ago

The generation in gen_model_answer seems always using do_sample=False (GenerationConfig's default). It invalidates configs on top_p, top_k, etc.

passaglia commented 10 months ago

You're correct do_sample was being ignored, thank you very much for pointing this out! I've pushed a fix now.

Since GenerationConfig defaults do_sample to False, all local models used in the ranking so far were generated with greedy sampling (except RWKV, which uses a separate generate function). I don't think this is bad -- in fact I think it is good for reproducibility, and in a future update I think I will set greedy_sampling by default