search

abetlen / llama-cpp-python

Python bindings for llama.cpp
https://llama-cpp-python.readthedocs.io
MIT License
7.71k stars 928 forks source link

[Missing documentation/functionality]: choices #1130

Open BlackLotus opened 7 months ago

BlackLotus commented 7 months ago

When using llama-cpp-python the dict entry choices is always created. I couldn't find any documentation on what choices stands for. While skimming the code it looks like there is always only one list entry created. I couldn't find any example or documentation to the choices entry. Why is it called choices? How can I generate more choices?

Depending on the reason why it's called choices I would like to propose an implementation for it. If it's called choices, because of historical reasons (meaning there is no real rhyme or reason) it would be nice to generate multiple possible responses to the same prompt. You could specify that you want to generate 5 responses and then let some heuristic go over the choices and present the best one/

If there is a reason why it's called choices and a possibility to have more than one choice I would be interested in that as well.

abetlen commented 7 months ago

Hey @BlackLotus you're exactly right, the choices key actually comes from the OpenAI API but it's unnused in this library at the moment. I'm currently working on the multi-completion feature you describe (implemented through the n parameter of the completions API and batch processing) and once that's complete the choice object will be used to return multiple completions to the same base prompt.