search

liltom-eth / llama2-webui

Run any Llama 2 locally with gradio UI on GPU or CPU from anywhere (Linux/Windows/Mac). Use `llama2-wrapper` as your local llama2 backend for Generative Agents/Apps.
MIT License
1.96k stars 201 forks source link

[FEATURE] Is there any way to expose this as Rest API instead of Default UI #27

Closed ajasingh closed 11 months ago

ajasingh commented 1 year ago

Is there any way where this could be exposed as Rest API instead of a WebUI default

liltom-eth commented 1 year ago

Yeah. A REST API should be helpful to be served as a backend for other AI applications. Still working on that.

ruizcrp commented 11 months ago

This feature would be great! Is there still development on it?

liltom-eth commented 11 months ago

@ruizcrp This feature has already been implemented. Check the doc here.

ruizcrp commented 11 months ago

Hi, thanks for the answer. I saw that but thought that both cannot run at the same time. E.g. two different processes that would require double the RAM if they are running at the same time and could not share a common queue. But maybe that's not the case?

liltom-eth commented 11 months ago

@ruizcrp Yeah, thanks for letting me know about that use case. The API is implemented later after the UI. Maybe you can try some OpenAI-compatible UI since our API is also OpenAI-compatible.