search

openlm-research / open_llama

OpenLLaMA, a permissively licensed open source reproduction of Meta AI’s LLaMA 7B trained on the RedPajama dataset
Apache License 2.0
7.29k stars 372 forks source link

compatibility with llama.cpp #38

Closed nivibilla closed 1 year ago

nivibilla commented 1 year ago

Hey, first of all, amazing work. Thanks for building an open llama model. As the title suggests, I would like to know if this model could be compatible with llamacpp

Thanks

codesoap commented 1 year ago

Works flawlessly with OpenAlpaca and I strongly assume this means OpenLLaMa will work as well.

Check out https://github.com/ggerganov/llama.cpp/issues/1291.

nivibilla commented 1 year ago

Amazing, then we can take advantage of all the tools in llama.cpp. Especially the M1 gpu inference. I get 88ms per token on guanaco 13b on my MacBook. which is quite fast considering how big the model is.