search

anon998 / simple-proxy-for-tavern

GNU Affero General Public License v3.0
110 stars 6 forks source link

llama.cpp server compatibility #11

Open MeeTee55 opened 1 year ago

MeeTee55 commented 1 year ago

Hi! Simple proxy is working fine with the updated llama.cpp server right now (good work btw!) but I suspect it is not fully compatible, since it was developed before the server update. Can I just add the missing parameters like stream etc to llamaCppSettings: {} in config.mjs, or it is more complex than that? Sorry, have no idea how node works, lol.

anon998 commented 1 year ago

It was updated at the same time because I was contributing to the llama.cpp server PR too. It already includes the stream parameter, you just have to enable streaming on Silly.

https://github.com/anon998/simple-proxy-for-tavern/blob/d14f43a884418e5297213a55aee0d9b8694a6efe/src/backends.mjs#L132-L134

You can use llamaCppSettings to override or add any setting.

You can add these ones to the generation preset too, although they only will work for llama.cpp:

    presence_penalty: 
    frequency_penalty: 
    mirostat: 
    mirostat_tau: 
    mirostat_eta:
MeeTee55 commented 1 year ago

Perfecto! Many thanks!