search

go-skynet / go-llama.cpp

LLama.cpp golang bindings
MIT License
614 stars 78 forks source link

Is there any example code base using this binding #297

Open cmingxu opened 7 months ago

cmingxu commented 7 months ago

I am wondering is there any project already use this project?

ChinaGISboyYang commented 7 months ago

me too

synw commented 7 months ago

There is my inference server project: https://github.com/synw/goinfer

RobinHeitz commented 3 months ago

@synw Sorry for off-topic question, but you might have some experience:

I'm testing around with LLama2 models, and I found that it's extremely slow, especially if there is a bit of context in the prompt. At the beginning it was at full workload (CPU wise), now its around 10-15% and the prediction takes like 30 mins. I assume it looks different to you, right?

synw commented 3 months ago