search

3x3cut0r / llama-cpp-python-streamlit

A streamlit app for using a llama-cpp-python high level api
MIT License
9 stars 3 forks source link

not recieving back from llama.cpp #1

Closed zooteld closed 11 months ago

zooteld commented 11 months ago

2023-11-03_130748

It appears its not receiving the response back from the llama.cpp server. please advise

zooteld commented 11 months ago

*I tested with the index.html that is included with llama.cpp and everything works as expected.

3x3cut0r commented 11 months ago

Hi, first of all, this app is far away of been used in production and still under development.

I tryed it with my own llama-cpp-python docker container and it is working fine.

did you put your llama-api address into src/config.json? did you test your api using postman or something else to check if it is running fine? for example can you reach http://<your-api-ip>:<port>/docs ?

br

zooteld commented 11 months ago

Let me be clear that I am not complaining, but asking for help.

did you put your llama-api address into src/config.json? did you test your api using postman or something else to check if it is running fine? for example can you reach http://:/docs ?

Yes, yes, and no... I am using a local hosted llama.cpp server with amd cards I tested the server with the index.html that comes with it in its examples and it is working nicely but when i try to use this beautiful app i can see it hitting the server in the server console but its not assigned a task like the ones from the index.html.

to me it looks like the request might not be the correct format.

3x3cut0r commented 11 months ago

Ok i got it. It is because you are not using llama-cpp-python, but llama.cpp. the projects have different endponts, see here. i have not implemented the /completion endpoint yet and this is the only endpoint that is the same in both projects. if i have time, i will implement switching endpoints soon. currently it is set to /v1/chat/completion

zooteld commented 11 months ago

yes sir, I changed the /v1/chat/compleations to just compleation. it is connecting but after that it looses me

zooteld commented 11 months ago

my repo synced lol and now im getting a "no context to append"

3x3cut0r commented 11 months ago

the problem is the other endpoint uses differend (json-)structure. by just switching the endpoint address it will not gonna work.

zooteld commented 11 months ago

ok thank you for this information. I will attempt to reasearch further into changing the endpoint json format, and maybe i will stumble across something. sorry if I bother you.

3x3cut0r commented 11 months ago

I updated the app and added a endpoint dropdown to select other endpoints. Try using /completions to use llama.cpp endpoint. tell me if its working or if there are some bugs.

zooteld commented 11 months ago

That's awesome! yes I got it to connect and work. I had to use the api_like_OAI.py on the server and /v1/completions in the app. nice app!