Support GPT4All Server API

ThiloteE commented 2 days ago

Describtion of solution:

I want to use JabRef's AI feature locally. There are multiple applications out there that provide a server API. They very often offer an API that resembles OpenAI API.

GPT4All is such an application. Others are Llama.cpp, Ollama, LMStudio, Jan, KobolCPP. I am sure, there are more, but those are the most well known ones.

The grand advantage of those applications is that they offer more samplers, GPU acceleration, hardware support and support for models that have not been added to JabRef.

Problem

It kinda works with GPT4All already, but something is wrong. I believe the embeddings are not sent together with the prompt and responses look like they are cutoff in the middle.

GPT4All:

JabRef:

Additional context

GPT4All documentation for local API server
The phi-3.1-mini-128k-instruct model can be downloaded here: https://huggingface.co/GPT4All-Community/Phi-3.1-mini-128k-instruct-GGUF/tree/main. Just move the model file into the model directory of GPT4All and then configure it in the model settings as shown in my screenshots down below.
Documentation about how to configure other custom models: https://github.com/nomic-ai/gpt4all/wiki/Configuring-Custom-Models.

JabRef preferences:

GPT4All preferences:

GPT4All model settings 1:

GPT4All model settings 2:

ThiloteE commented 2 days ago

I also often get those errors/warnings in the commandline, when I try to send messages when connected to GPT4All server API. Not sure, if related.

2024-10-01 13:13:53 [pool-2-thread-4] org.jabref.logic.ai.chatting.AiChatLogic.execute()
INFO: Sending message to AI provider (https://api.openai.com/v1) for answering in entry CooperEtAl200708cah: What are the authors of the paper?
2024-10-01 13:13:53 [JavaFX Application Thread] org.jabref.gui.ai.components.aichat.AiChatComponent.lambda$onSendMessage$11()
ERROR: Got an error while sending a message to AI: io.github.stefanbratanov.jvm.openai.OpenAIException: 400 - message: Invalid 'messages[2].role': did not expect 'user' here, type: invalid_request_error, param: null, code: null
        at jvm.openai@0.11.0/io.github.stefanbratanov.jvm.openai.OpenAIClient.lambda$validateHttpResponse$6(OpenAIClient.java:129)
        at java.base/java.util.Optional.ifPresentOrElse(Optional.java:196)
        at jvm.openai@0.11.0/io.github.stefanbratanov.jvm.openai.OpenAIClient.validateHttpResponse(OpenAIClient.java:127)
        at jvm.openai@0.11.0/io.github.stefanbratanov.jvm.openai.OpenAIClient.sendHttpRequest(OpenAIClient.java:85)
        at jvm.openai@0.11.0/io.github.stefanbratanov.jvm.openai.OpenAIClient.sendHttpRequest(OpenAIClient.java:78)
        at jvm.openai@0.11.0/io.github.stefanbratanov.jvm.openai.ChatClient.createChatCompletion(ChatClient.java:37)
        at org.jabref@100.0.0/org.jabref.logic.ai.chatting.model.JvmOpenAiChatLanguageModel.generate(JvmOpenAiChatLanguageModel.java:65)
        at org.jabref@100.0.0/org.jabref.logic.ai.chatting.model.JabRefChatLanguageModel.generate(JabRefChatLanguageModel.java:142)
        at langchain4j@0.34.0/dev.langchain4j.chain.ConversationalRetrievalChain.execute(ConversationalRetrievalChain.java:85)
        at langchain4j@0.34.0/dev.langchain4j.chain.ConversationalRetrievalChain.execute(ConversationalRetrievalChain.java:32)
        at org.jabref@100.0.0/org.jabref.logic.ai.chatting.AiChatLogic.execute(AiChatLogic.java:168)
        at org.jabref@100.0.0/org.jabref.gui.ai.components.aichat.AiChatComponent.lambda$onSendMessage$9(AiChatComponent.java:204)
        at org.jabref@100.0.0/org.jabref.logic.util.BackgroundTask$1.call(BackgroundTask.java:73)
        at org.jabref@100.0.0/org.jabref.gui.util.UiTaskExecutor$1.call(UiTaskExecutor.java:191)
        at javafx.graphics@23/javafx.concurrent.Task$TaskCallable.call(Task.java:1401)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)

ThiloteE commented 2 days ago

I think there is also an issue when switching models. Even after clearing the chat history, it is not possible to get around this erorr message, unless switching to a different entry in JabRef. Then I can chat again.

ThiloteE commented 2 days ago

@InAnYan

InAnYan commented 2 days ago

Such an interesting issue and such an interesting behaviour...

IMHO it's very wrong that there is no OpenAI API compatible mode in gpt4all. (Standardization made people's life much easier).

However, I will look into this issue in more detail, because Gpt4All is a popular app and this is also important

JabRef / jabref