Closed h1alexbel closed 8 months ago
since deep infra aggregates different models, we should have deepinfra_endpoint
as well. For Phind-CodeLlama-34B-v2 it should be https://api.deepinfra.com/v1/inference/Phind/Phind-CodeLlama-34B-v2
, but we can try to use Open AI compatible API, its here, using this endpoint will be the same for all models ChatGPT like: https://api.deepinfra.com/v1/openai/chat/completions
using Open AI endpoint we can pass model
as we are doing now for openai_model
input, for Phind-CodeLlama-34B-v2 it will be "model": "Phind/Phind-CodeLlama-34B-v2",
curl "https://api.deepinfra.com/v1/openai/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $(deepctl auth token)" \
-d '{
"model": "Phind/Phind-CodeLlama-34B-v2",
"messages": [
{
"role": "user",
"content": "Hello!"
}
]
}'
Let's support deep infra models, like Phind/Phind-CodeLlama-34B-v2, we can use it via Deep Infra API
We should define another input called
deepinfra_token
, and if this input is present, ask the Deep Infra chatbot, but if it does not, then we trying withopenai_token