Registering additional local models

[psychemedia]

Is there a simple way of registering additional local models ?

For example, a templated/cookiecutter Python package that could be used to specify the model name and path?

It would also be useful to be able to register local embeddings models.

[3coins]

@psychemedia We don't have a cookiecutter template for adding a new provider at this time. But here is some info on adding new providers/models. You can add a new provider (and models) by creating a python package exposing an entry point within the group jupyter_ai.model_providers or jupyter_ai.embeddings_model_providers. The provider class should extend the BaseProvider or BaseEmbeddingsProvider. One more thing to note is we rely on LangChain to orchestrate LLM calls, so any provider you add should have an integration available in LangChain.

[DanielCastroBosch]

@JasonWeill , did you manage to load a offline local model file? I need it too. Thanks in advance....

[JasonWeill]

@DanielCastroBosch You can load local models with GPT4All and Ollama for use with Jupyter AI. See the docs: https://jupyter-ai.readthedocs.io/en/latest/users/index.html#gpt4all-usage-early-stage

[DanielCastroBosch]

@JasonWeill Thanks ! I have my own llm server with a simple interface. It works good on python but It´s possible to add my own model to the jupyter-ai chat configuration drop down box ? Is there any link to help me on that ? Something like Myserver::MyModel?

[krassowski]

It´s possible to add my own model to the jupyter-ai chat configuration drop down box ? Is there any link to help me on that ?

Yes, https://jupyter-ai.readthedocs.io/en/latest/developers/index.html#custom-model-providers

[DanielCastroBosch]

Hi @krassowski ! Thanks for the answer. I did like the model in the link :

from jupyter_ai_magics import BaseProvider
from langchain_community.llms import FakeListLLM
import requests
import json
import urllib3

class MyProviderChat(BaseProvider, FakeListLLM):
    id = "myProvider"
    name = "PROVIDER"
    model_id_key = "model"
    models = ["provider"]
    message=""
    token_url = "<TOKEN URL>"
    client_id = "<CLIENT ID>"
    client_secret = "<CLIENT SECRET>"
    scope = "<scope>"
    TryoutBaseURL = "BASE URL API"
    proxy = { 'http': '<HTTP PROXY>', 'https': '<PROXY HTTPS>' }
    urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

    def get_access_token():
        data = {
        "client_id": client_id,
        "scope": scope,
        "client_secret": client_secret,
        "grant_type": "client_credentials",
     }
        response = requests.post(token_url, data=data, verify=False, proxies=proxy)
        response_data = response.json()
        return response_data.get("access_token")

    def send_message_to_llm(user_message):
        # Get the access token
        access_token = super().get_access_token()

        # Set up the headers
        headers = {
         "Content-Type": "application/json",
         "Authorization": f"Bearer {access_token}",
         "ocp-Apim-Subscription-Key": "<api key>", 
        }

        # Set up the payload
        query_data = {
         "messages": [
            {
                "role": "user",
                "content": f"{user_message}"
            }
        ],
         "model": "gpt3",
         "temperature": 0.5
        }

        # Send POST request
        response = requests.post(TryoutBaseURL, headers=headers, json=query_data, verify=False, proxies=proxy)

        # Check if the request was successful
        if response.status_code == 200:
           # Parse the JSON response
           result = response.json()

           # Print the assistant's response
           for message in result:
             if message["role"] == "assistant":
                print(message["content"])
                #print("Token count:", message["tokenCount"])
                #print("Token limit exceeded:", message["tokenLimitExceeded"])
        else:
           print("Error:", response.status_code, response.text)

        def __init__(self, **kwargs):

            message = kwargs.get("message")
            msg = super().send_message_to_llm(message)
            super().__init__(**kwargs)

But I am getting the following error:

Traceback (most recent call last):
  File "/opt/python/3.12.0/lib/python3.12/site-packages/jupyter_ai/chat_handlers/base.py", line 181, in on_message
    await self.process_message(message)
  File "/opt/python/3.12.0/lib/python3.12/site-packages/jupyter_ai/chat_handlers/default.py", line 96, in process_message
    self.get_llm_chain()
  File "/opt/python/3.12.0/lib/python3.12/site-packages/jupyter_ai/chat_handlers/base.py", line 330, in get_llm_chain
    self.create_llm_chain(lm_provider, lm_provider_params)
  File "/opt/python/3.12.0/lib/python3.12/site-packages/jupyter_ai/chat_handlers/default.py", line 38, in create_llm_chain
    llm = provider(**unified_parameters)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/python/3.12.0/lib/python3.12/site-packages/jupyter_ai_magics/providers.py", line 348, in __init__
    super().__init__(*args, **kwargs, **model_kwargs)
  File "/opt/python/3.12.0/lib/python3.12/site-packages/pydantic/v1/main.py", line 341, in __init__
    raise validation_error
pydantic.v1.error_wrappers.ValidationError: 1 validation error for MyProviderChat
responses
  field required (type=value_error.missing)

Can you help me please?

[krassowski]

@DanielCastroBosch The error says that FakeListLLM expects responses argument but you are not providing it. I think that you are treating the example too literally. The FakeListLLM is just an example, you should remove it in your final usage, as the name says it is "Fake". The docs currently say:

Custom providers inherit from both jupyter-ai’s BaseProvider and langchain’s LLM. You can either import a pre-defined model from LangChain LLM list, or define a custom LLM. In the example below, we define a provider with two models using a dummy FakeListLLM model, which returns responses from the responses keyword argument

Since this is not the first time I see someone make this mistake, would you mind suggesting a better wording and opening a PR with clarification?