search

cogentapps / chat-with-gpt

An open-source ChatGPT app with a voice
https://www.chatwithgpt.ai
MIT License
2.31k stars 493 forks source link

Support for ggerganov/llama.cpp #17

Open ricklamers opened 1 year ago

ricklamers commented 1 year ago

Would be cool if it could support https://github.com/ggerganov/llama.cpp

Is it on your roadmap?

ricklamers commented 1 year ago

Maybe using https://cocktailpeanut.github.io/dalai/#/

cogentapps commented 1 year ago

Integrating with the Dalai API should be doable.

How well does Llama perform in a chat style / conversational format?

schmidp commented 1 year ago

with https://alpaca-ai-custom5.ngrok.io it seems to perform well

rounakdatta commented 1 year ago

Alpaca is explicitly trained for Instruct-style messages. LLAMA by default isn't ready for conversations (yet) without some clever prompt engineering.

cogentapps commented 1 year ago

I'd like to implement this via Dalai's API which now supports Alpaca, but this issue needs resolved first:

https://github.com/cocktailpeanut/dalai/issues/51

It's also a bit slow for longer chats. Ideally it would use llama.cpp's interactive mode instead of resending the whole context with each message, but I don't think this is implemented yet?