Closed toshiakit closed 1 month ago
@jonasendc provided this comment:
Hi @toshiakit , maybe something like:
client = openai.AzureOpenAI(
api_version="2024-03-01-preview",
azure_endpoint="",
api_key=api_key,
)
Where azure_endpoint is some other url than openAIs. So somewhere in this app the URL is hardcoded. This should be dynamic.`
@Mingzefei provided this comment:
Hi, sorry for the late reply. @jonasendc has provided a case for AzureOpenAI, and I'd like to add another case about locally deployed LLMs.
For example, projects like olloma allow for easy local deployment and utilization of many open-source LLMs. After a successful deployment, olloma by default starts an API service on local port 11434. This API service is quite similar to calling the OpenAI API, as shown below:
from openai import OpenAI
client = OpenAI(
base_url = 'http://localhost:11434/v1',
api_key='ollama', # required, but unused
)
response = client.chat.completions.create(
model="llama3",
messages=[
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Who won the world series in 2020?"},
{"role": "assistant", "content": "The LA Dodgers won in 2020."},
{"role": "user", "content": "Where was it played?"}
]
)
print(response.choices[0].message.content)
As you can see, generally only the base_url needs to be modified.
Additionally, being able to call ollama or other locally deployed LLMs might have the following advantages:
The ability to use LLMs fine-tuned for specific tasks like MATLAB, which might yield better performance and lower costs than ChatGPT.
Providing a solution for countries or regions where ChatGPT is not available.
I hope this adds to the discussion and look forward to your thoughts.
A temporary solution is as follows: 1) Just open this file +llms/+internal/callOpenAIChatAPI.m 2) change END_POINT to your Custom API Endpoint in this line: END_POINT = "https://api.openai.com/v1/chat/completions";
That should be OK for Openai-style LLMs.
As noted in the PR linked above, that may or may not work with any allegedly OpenAI-compatible endpoint. A little more discovery is needed to decide how well we can support “arbitrary” end points in general.
With the Azure and Ollama support now in main
(including the little details they do need to be different), and support for non-local Ollama servers coming in #53, I'm closing this request. Experience with the endpoints currently supported shows that only a subset of functionality could be supported uniformly and the data formats are different in practice.
per https://github.com/toshiakit/MatGPT/issues/30
This refers to the use of custom assistants https://community.openai.com/t/custom-assistant-api-endpoints/567107