Use generic OpenAI API for local LLM (ooba text-gen, LocalAI, llama-cpp-python , vllm , etc)

[neowisard]

Hi ! i use standalone server with ooba's textgen with Open AI generic API and token . And local fast whisper. Few apps for OpenAI works with api perfectly, and now i can try connect your integration. What i doing wrong ? This error originated from a custom integration. Tested with\withot auth\token, ver 0.11, 1.0.beta, 1.0, 1.0.1 beta.

Logger: custom_components.extended_openai_conversation Source: custom_components/extended_openai_conversation/init.py:189 Integration: Extended OpenAI Conversation (documentation, issues) First occurred: 4:35:02 PM (7 occurrences) Last logged: 6:33:29 PM

Internal Server Error

FYI: The best implementation of the interface with openapi I've seen in chatbot ( with mongo as storage, whisper and pictures ) https://github.com/father-bot/chatgpt_telegram_bot/blob/main/bot/openai_utils.py

UPD: This implementation talk\speak via API , but can't control \ functions https://github.com/drndos/hass-openai-custom-conversation

[jekalmin]

Thanks for reporting an issue!

As far as I know, text-generation-webui doesn't support function calling. Although LocalAI supports function calling, there are still many issues to be resolved.

Please refer https://github.com/jekalmin/extended_openai_conversation/issues/17 and https://github.com/jekalmin/extended_openai_conversation/issues/85 about LocalAI issues.

[neowisard]

Yep, in ooba's textgen functions does not work yet. check it :

curl --location 'http://192.168.0.55:5000/v1/chat/completions' \
  -H 'Authorization: Bearer toofoo' \
  -H 'Content-Type: application/json' \
  --data '{
     "messages": [
        {
            "role": "system",
            "content": "You are my smart home assistant."
        },
        {
            "role": "user",
            "content": "turn on livingroom light"
        }
    ],
    "functions": [
        {
            "name": "execute_services",
            "description": "Execute service of devices in Home Assistant.",
            "parameters": {
                "type": "object",
                "properties": {
                    "domain": {
                        "description": "The domain of the service.",
                        "type": "string"
                    },
                    "service": {
                        "description": "The service to be called",
                        "type": "string"
                    },
                    "service_data": {
                        "type": "object",
                        "properties": {
                            "entity_id": {
                                "type": "array",
                                "items": {
                                    "type": "string",
                                    "description": "The entity_id retrieved from available devices. It must start with domain, followed by dot character."
                                }
                            }
                        },
                        "required": [
                            "domain",
                            "service"
                        ]
                    }
                }
            }
        }
    ],
    "function_call": "auto",
    "temperature": 0.7
}'

and https://platform.openai.com/docs/api-reference/chat/create

Openai has deprecated previous function calling api key phrases and replace by tools

[neowisard]

I honestly tried to run this locally. You made a great plugin and it works well with OpenAI, locally unfortunately I was not able to run any functions with or without tools. Maybe if someone, even one person will have success and he will post here how to do it. i tried next APIs with mixtral, mistral, functionary models

• LocalAI - talk and not execute(ignore) any function
• llama-cpp_python - talk and not execute (ignore) any function
• ooboga textgen - didnt have functions. Only talk (remove functions code to [])
• vllm and funtionary models (cant start on Tesla P40 GPU)