BaseUri is intentionally made for user to include /v1/, check out the use of openai library in python:
from openai import OpenAI
client = OpenAI(
base_url = 'http://localhost:11434/v1',
api_key='ollama', # required, but unused
)
compare:
val client = OpenAISyncClient("ollama", uri"http://localhost:11434/v1")
Things left to figure out:
Lots of APIs do not work, most of the time what does work is just /v1/chat/completions, /v1/completions and if weather, moon phase, time of day and horoscopes are right - /v1/embeddings too. Ollama doesn't implement /v1/models but Grok does and so on.
Error modes are slightly different. When other companies / foss orgs say that API is openAI-compatible it doesn't mean it handles errors the same way. For example: ollama behaves fine and if you use a non-existent model you get a nice:
sttp.openai.OpenAIExceptions$OpenAIException$InvalidRequestException: statusCode: 404, response: {"error":{"message":"model 'mixtral-8x7b-32768' not found, try pulling it first","type":"api_error","param":null,"code":null}}
which is sort of what I'd expect (and what sttp-openai would expect too). For Grok however that's not as nice because their error json doesn't contain one of the expected fields, apparently:
scala> println(oai.createChatCompletion(ChatBody(model = ChatCompletionModel.CustomChatCompletionModel("mistral"), messages = Seq(Message.UserMessage(content = Content.TextContent("What is Scala programming language?"))))).choices.head.message.content.trim)
upickle.core.Abort: missing keys in dictionary: param
λ curl https://api.groq.com/openai/v1/chat/completions -w "%{http_code}" \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer REDACTED' \
-d '{
"model": "mistral",
"messages": [
{
"role": "user",
"content": "How do I shot web?"
}
]
}'
{"error":{"message":"The model `mistral` does not exist or you do not have access to it.","type":"invalid_request_error","code":"model_not_found"}}
404
Now this is probably another issue with upickle because param field is usually defined as param: Option[String] and when it's not present in the json is should be deserialized to None. Maybe we should configure upickle to be more permissive in error json parsing?
Closes #166
BaseUri is intentionally made for user to include
/v1/
, check out the use ofopenai
library in python:compare:
Things left to figure out:
/v1/chat/completions
,/v1/completions
and if weather, moon phase, time of day and horoscopes are right -/v1/embeddings
too. Ollama doesn't implement/v1/models
but Grok does and so on.which is sort of what I'd expect (and what
sttp-openai
would expect too). For Grok however that's not as nice because their error json doesn't contain one of the expected fields, apparently:Now this is probably another issue with upickle because
param
field is usually defined asparam: Option[String]
and when it's not present in the json is should be deserialized to None. Maybe we should configure upickle to be more permissive in error json parsing?