Open Dan-wanna-M opened 4 weeks ago
@AllentDan May consider this feature in PyTorchEngine. cc @grimoire
@Dan-wanna-M I am afraid we cannot support this feature in TurboMindEngine, since it is developed by C++ and CUDA.
An external python logits_processors
function can't be passed to it.
@Dan-wanna-M I am afraid we cannot support this feature in TurboMindEngine, since it is developed by C++ and CUDA. An external python
logits_processors
function can't be passed to it.
Got it. It would also be great if we can have some kinds of interface that allows us to do native function callback, since libraries implemented in C/C++/rust can leverage that feature.
Motivation
Many
lmdeploy
counterparts(vllm, transformers, exllamav2...) providelogits_processors
that allow users to modify the logits before softmax. This enables many useful features like constrained decoding, classifier-free guidance, custom repetition penalty, etc.Related resources
Huggingface API: https://huggingface.co/docs/transformers/en/internal/generation_utils#logitsprocessor Vllm API: https://github.com/vllm-project/vllm/issues/1728
Additional context
I wrote a constrained decoding library and is planning to integrate
lmdeploy
into my library.