search

regiellis / civitai-models-cli

Simple CLI tool streamlines the process of managing AI models from the CivitAI platform. It offers functionalities to list available models, view their details, download selected variants, and remove models from local storage and provides a summary of the model description using Ollama or OpenAI.
MIT License
3 stars 2 forks source link

OpenAi compatible APi endpoint #1

Open ali0une opened 3 months ago

ali0une commented 3 months ago

Hi there.

Very nice project!

Would it be possible to use an OpenAi compatible APi endpoint with a local LLM through LM Studio or text-generation-webui or llama.cpp?

i see llama.cpp has API Endpoints and even OAI-like API but i've never used them.

But i use LM Studio in server mode and it can expose an Open Api compatible endpoint like http://127.0.0.1:5000/v1

i'm pretty sure it's the same for text-generation-webui.

regiellis commented 3 months ago

Hi there.

Very nice project!

Would it be possible to use an OpenAi compatible APi endpoint with a local LLM through LM Studio or text-generation-webui or llama.cpp?

i see llama.cpp has API Endpoints and even OAI-like API but i've never used them.

But i use LM Studio in server mode and it can expose an Open Api compatible endpoint like http://127.0.0.1:5000/v1

i'm pretty sure it's the same for text-generation-webui.

Hey, I looked into this...because I am using the official python lib for my local calls...this should be supported out of the box. Just update the .env file to point to the http://[IP]:[PORT] and update the model you want to use. Should all work; I will be doing a test later to close this out...but you should have no issue as long as it is openAI compatible

ali0une commented 3 months ago

i'll have a look and let you know. Thanks!

ali0une commented 3 months ago

So here my .env : 

#https://developer.civitai.com/docs/getting-started/setup-profile#create-an-api-key
CIVITAI_TOKEN=xxxx
#/location/of/your/models/
MODELS_DIR=/whatever/models/

# http://localhost:11434 or http://host.docker.internal:11434
OLLAMA_API_BASE=http://localhost:5000/v1/ # same with http://localhost:5000
# Tuned to tinydolphin..as best I could
OLLAMA_MODEL=Meta-Llama-3.1-8B-Instruct
HTML_OUT=False # Some models require HTML conversion, some don't
TEMP=0
TOP_P=0

#OPENAI_MODEL=gpt-4o-mini
#OPENAI_API_KEY= # sk-proj

#GROQ_MODEL=llama-3.1-70b-versatile
#GROQ_API_KEY=#//

At launch the LM Studio server print this : 


[2024-08-11 14:03:50.857] [INFO] [LM STUDIO SERVER] Verbose server logs are ENABLED
[2024-08-11 14:03:50.860] [INFO] [LM STUDIO SERVER] Success! HTTP server listening on port 5000
[2024-08-11 14:03:50.860] [INFO] [LM STUDIO SERVER] Supported endpoints:
[2024-08-11 14:03:50.860] [INFO] [LM STUDIO SERVER] ->  GET  http://localhost:5000/v1/models
[2024-08-11 14:03:50.860] [INFO] [LM STUDIO SERVER] ->  POST http://localhost:5000/v1/chat/completions
[2024-08-11 14:03:50.860] [INFO] [LM STUDIO SERVER] ->  POST http://localhost:5000/v1/completions
[2024-08-11 14:03:50.860] [INFO] [LM STUDIO SERVER] ->  POST http://localhost:5000/v1/embeddings     <------------ NEW!
[2024-08-11 14:03:50.860] [INFO] [LM STUDIO SERVER] Model loaded: local/Meta-Llama-3-8B-Instruct/Meta-Llama-3.1-8B-Instruct-Q5_K_M.gguf
[2024-08-11 14:03:50.860] [INFO] [LM STUDIO SERVER] Logs are saved into /tmp/lmstudio-server-log.txt

And this if i issue a request with civitai-cli-manager and the .env above :

[2024-08-11 14:04:23.335] [ERROR] Unexpected endpoint or method. (POST /api/chat). Returning 200 anyway

i think we would need to have a specific configuration for openAi compatible APi like textgeneration-webui or LM Studio

regiellis commented 3 months ago

@ali0une Strange...it should work as they both follow the OpenAI Standard...I will need to install LLMStudio server on my server and tested against it.