Closed caioxapelao closed 1 month ago
OK, so I had an issue releated to the API yesterday and I found 1 thing, if you don't include the POST method you get an error 404. Since you don't include a log, detailed description of the error and the endpoint used this may or may not be related.
URI = http://x.x.x.x:4891/v1/completions Request body: "model" = "text-davinci-003" (filename of your model minus the extension so bla-model.gguf becomes bla-model) "prompt" = $query "max_tokens" = 500 "temperature" = 0.5
Request headers: Method = Post Contenttype = application/json (not sure if this is re quired)
I can assure you it is working. So I'm going to turn this into a question instead. See also:
Note: it is only accessible from localhost
aka 127.0.0.1
and not through https
, only http
.
The easiest way to get a response is querying the list of models: http://localhost:4891/v1/models. This is a GET
request, so you can easily do it in your browser.
Afterwards, you're most likely interested in talking to /v1/chat/completions
, which is a POST
request. Note that you need to somewhat follow the OpenAI API spec when querying the server. It's not a complete implementation, however.
As you're on Windows, here's an example with PowerShell using the model Phi-3 Mini Instruct
:
Invoke-WebRequest -URI http://localhost:4891/v1/chat/completions -Method POST -ContentType application/json -Body '{
"model": "Phi-3 Mini Instruct",
"messages": [{"role":"user","content":"hi, who are you?"}],
"max_tokens": 2048,
"temperature": 0.7
}'
For others, here's the same example with cURL instead:
curl -X POST http://localhost:4891/v1/chat/completions -d '{
"model": "Phi-3 Mini Instruct",
"messages": [{"role":"user","content":"hi, who are you?"}],
"max_tokens": 2048,
"temperature": 0.7
}'
This should be clear enough for you to get it to work, right?
This issue is incomplete. Please provide additional information. What did you try, what did you expect to happen, and what actually happened? There are a few pitfalls to the local server that are not obvious from within the chat application, and we could probably document them better.
Closing due to lack of response, feel free to reopen if you have anything to add.
Bug Report
GPT4All Local server not working
Steps to Reproduce
Expected Behavior
Local OpenAI API Endpoint
Your Environment