[Feature] Multi options

[QinghanLai]

Checklist

• [ ] 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• [ ] 2. Please use English, otherwise it will be closed.

Motivation

I want to use sgl.gen to select multiple options from the candidate selection, does it support it?

Related resources

No response

[ByronHsu]

Do you mean select like top K choices from the candidates?

[QinghanLai]

No!

my mean is given an option list, eg: [A,B,C,D,E] it can select multi option in llm decode, such as [A,C,D] or [D,B] or [D,A,E] ....., not using top-k selection

Qinghan Lai Qilu University Of Technology(Shangdong Academy Of Sciences) On 10/23/2024 10:07，Byron @.***> wrote：

Do you mean select like top K choices from the candidates?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[QinghanLai]

This multi option is very commonly used in our development tasks.

The top k method will destroy the original output logic of the LLM and reduce the effect of generating text.

I'd love to see sglang support that, as he's really helpful for tasks and can compensate for the limitations of single choice.

Thanks!

Qinghan Lai Qilu University Of Technology(Shangdong Academy Of Sciences) On 10/23/2024 10:21，Qinghan @.***> wrote： No!

my mean is given an option list, eg: [A,B,C,D,E] it can select multi option in llm decode, such as [A,C,D] or [D,B] or [D,A,E] ....., not using top-k selection

Qinghan Lai Qilu University Of Technology(Shangdong Academy Of Sciences) On 10/23/2024 10:07，Byron @.***> wrote：

Do you mean select like top K choices from the candidates?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>