Closed wsxiaoys closed 3 months ago
I've started working on this. I'll have a draft CL with the implementation to make sure I have the logic right. Might need some help on what the interface changes to TextInferenceEngine
will look like to make this possible.
Also, right now, the decoding in entirely greedy, should I continue to use greedy decoding for the speculative model as well?
Thanks for claiming the feature!
Also, right now, the decoding in entirely greedy, should I continue to use greedy decoding for the speculative model as well?
Yes - greedy decoding shall be good for now
Code location: https://github.com/TabbyML/tabby/blob/main/crates/llama-cpp-bindings/src/engine.cc Reference: https://github.com/ggerganov/llama.cpp/blob/master/examples/speculative/speculative.cpp#L47
Implement speculative decoding to speed up certain models.