search

go-skynet / go-llama.cpp

LLama.cpp golang bindings
MIT License
650 stars 79 forks source link

feat(speculative-sampling): add grammar support #203

Closed mudler closed 11 months ago

mudler commented 11 months ago

Adds support for: https://github.com/ggerganov/llama.cpp/pull/2991 (copying the required bits from the llama.cpp example)

part of https://github.com/go-skynet/LocalAI/issues/1013

Note from this version METAL is enabled by default on mac builds and as such needs to be disabled specifically in case it is not wanted (https://github.com/ggerganov/llama.cpp/pull/2901)