I add ollama backend support , after start ollama service , you can use "ollama:" prefix , etc. ollama:mistral:v0.3, ollama:llama3 , default ollama host is "http://localhost:11434"
curl http://localhost:8866/v1/chat/completions \
-X POST \
-H "Content-Type: application/json" \
-H "Authorization: Bearer br-a8OTgDNsdF0vYsevxtoEeVPAyze7P" \
-d '{
"model": "ollama:mistral:v0.3",
"messages": [
{
"role": "user",
"content": "who are you"
}
],
"stream": true,
"temperature": 0.1,
"max_tokens": 4096
}'
output:
{"choices":{"role":"assistant","content":" I am a model trained by Mistral AI. I was designed to assist with a wide range of tasks, answer questions, and engage in conversation on various topics. How can I help you today?"},"usage":{"completion_tokens":41,"prompt_tokens":7,"total_tokens":48}}
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.
I add ollama backend support , after start ollama service , you can use "ollama:" prefix , etc. ollama:mistral:v0.3, ollama:llama3 , default ollama host is "http://localhost:11434"
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.