search

guidance-ai / guidance

A guidance language for controlling large language models.
MIT License
19.09k stars 1.04k forks source link

[Feature Request] Support Kwargs in Gen #630

Open Hellisotherpeople opened 9 months ago

Hellisotherpeople commented 9 months ago

Is your feature request related to a problem? Please describe. I can't pass any parameters to the LLM generation function except the handful that you all have decided to support. That's insane when you consider the dozens and dozens of settings available in any huggingface model. This massively reduces the utility of this package for me.

Describe the solution you'd like Support for kwargs, maybe by implementing the paramater that you put in your code designed for this but left hanging

Harsha-Nori commented 8 months ago

Hey, this is totally fair. In its current implementation, guidance needs to take over the sampling loop from the LLM to enforce grammar compliance (and in part, for performance/efficiency). We'd definitely like to extend our sampler to support many of the underlying settings, but it's not entirely trivial to do this while staying both grammar-compliant and performant.

Can I ask which of the kwargs you're missing the most?

doomgrave commented 8 months ago

Hey, this is totally fair. In its current implementation, guidance needs to take over the sampling loop from the LLM to enforce grammar compliance (and in part, for performance/efficiency). We'd definitely like to extend our sampler to support many of the underlying settings, but it's not entirely trivial to do this while staying both grammar-compliant and performant.

Can I ask which of the kwargs you're missing the most?

I think repetition_penalty, frequency_penalty','presence_penalty' could be really useful to be customizable for each generation as we are able to do it with temperature.