zwilch
commented
6 months ago Can you use wireshark on loopback device to watch communication and can show what happenen on communication beetween client and gpt4all?
May this code examples help you:
node js
const OpenAI = require ('openai');
const openai = new OpenAI({
baseURL:'http://127.0.0.1:4891/v1',
apiKey: "not needed for a local LLM",
});
async function main() {
const text = await openai.chat.completions.create(
{
messages: [{ role: 'user', content: '}],
model: 'Nous Hermes 2 Mistral DPO',
max_tokens: 1024 ,
n: 1,
stop: null,
temperature: 0.35,
top_p:0.75,
stream:false,
},
{ maxRetries: 5,} )
console.log( text );
console.log( text.choices);
}//main
main();in Browser as fetch
const json_completion = JSON.stringify(
{stream:false,
temperatur:0.6,
max_tokens:100,
messages:[{role: "user", content:"Hello"},],
model: 'Nous Hermes 2 Mistral DPO'
}
);
const completions = await fetch("http://127.0.0.1:4891/v1/chat/completions",{
keepalive: true,
method: "POST",
mode: "no-cors",
// with this mode it will get a response ,
// but for security reason js in browser can not access the result of "await completions.json()"
headers: {
Accept: 'application/json',
'Content-Type': 'application/json',
'Access-Control-Allow-Origin': "*",
'Access-Control-Allow-Headers': "*"
},
body:json_completion
});
const completionjson = await completions.json();
/* here this is a problem,
* cause the browser can not do mode:"no-corse" and
* after finish the request do "completions.json()"
* this results in an error like
* Uncaught (in promise) SyntaxError: JSON.parse: unexpected end of data at line 1 column 1 of the JSON data
* see https://stackoverflow.com/questions/54896998/how-to-process-fetch-response-from-an-opaque-type
* without "mode:"no-corse" you will get an error like
* XHROPTIONS http://127.0.0.1:4891/v1/chat/completions CORS Preflight Did Not Succeed
* /
console.log(completionjson);
lrq3000
cosmic-snow
hyperstown
nebulous
xieu90
raja-jamwal
Bug Report
I tried to use GPT4All as a local LLM server with an OpenAI-like API for serving as a code copilot via the continue plugin for VSCode.
Unfortunately, whatever I tried, it did not work.
The server is correctly detected and all models are correctly loaded, using continue prerelease. However, when trying to send any message to gpt4all from continue, the response seems to be empty.
However, when I do my own curl query it works, so I don't know how to debug this further.
I have tried with Ollama and Koboldcpp (via the OpenAI-like API, same settings as for GPT4All - of course I changed the ports), and it worked for both flawlessly.
This seems to me to be an incompatibility in the API. Continue is expecting something that GPT4All is not providing or not in the expected format.
Steps to Reproduce
Expected Behavior
Continue should get non-empty responses from GPT4All.
Your Environment