Local LLMs

[LanceLake]

Allow to specify URL to use to run it with local LLMs like Ooga Booga.

[dlaliberte]

This works partially already. If you set your OAI_CONFIG_LIST.json file to contain something like this:

[
    {
        "model": "orca2",
        "api_key": "NULL",
        "base_url": "http://localhost:5001/v1"
    }
]

Then at least the autogen_test.py will use the given base_url. This is the default URL for the Kobold server. I don't think the model or api_key matter in this case. You have to start up the Kobold server with the model that will be used.

The example_rag.py program, which uses llama_index code, seems to have some hardwired use of the openai.com server. The example_research.py program makes use of the OpenAIWrapper, which also has hardwired use of openai's server. There might (or should) be enough configuration options via arguments to override the defaults, so we can use the same API software on the client side but have it communicate with our own server.

[LanceLake]

The python autogen_modified_group_chat.py and python autogen_standard_group_chat.py gives this error.

---

INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 401 Unauthorized" Traceback (most recent call last): File "E:\autogen-agi\autogen_modified_group_chat.py", line 86, in user_proxy.initiate_chat( File "C:\Python311\Lib\site-packages\autogen\agentchat\conversable_agent.py", line 544, in initiate_chat self.send(self.generate_init_message(context), recipient, silent=silent) File "C:\Python311\Lib\site-packages\autogen\agentchat\conversable_agent.py", line 344, in send recipient.receive(message, self, request_reply, silent) File "C:\Python311\Lib\site-packages\autogen\agentchat\conversable_agent.py", line 475, in receive reply = self.generate_reply(messages=self.chat_messages[sender], sender=sender) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\autogen\agentchat\conversable_agent.py", line 887, in generate_reply final, reply = reply_func(self, messages=messages, sender=sender, config=reply_func_tuple["config"]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\autogen-agi\autogen_mods\modified_group_chat.py", line 450, in run_chat speaker = groupchat.select_speaker(speaker, self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "E:\autogen-agi\autogen_mods\modified_group_chat.py", line 252, in select_speaker final, response = selector.generate_oai_reply(get_next_actor_message) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\autogen\agentchat\conversable_agent.py", line 619, in generate_oai_reply response = client.create( ^^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\autogen\oai\client.py", line 244, in create response = self._completions_create(client, params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\autogen\oai\client.py", line 314, in _completions_create response = completions.create(params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\openai_utils_utils.py", line 299, in wrapper return func(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\openai\resources\chat\completions.py", line 556, in create return self._post( ^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\openai_base_client.py", line 1055, in post return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\openai_base_client.py", line 834, in request return self._request( ^^^^^^^^^^^^^^ File "C:\Python311\Lib\site-packages\openai_base_client.py", line 877, in _request raise self._make_status_error_from_response(err.response) from None openai.AuthenticationError: Error code: 401 - {'error': {'message': 'Incorrect API key provided: sk-283827313. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}

python example_rag.py gives me

INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 401 Unauthorized" WARNING:llama_index.llms.openai_utils:Retrying llama_index.embeddings.openai.get_embeddings in 1.9060336807881066 seconds as it raised AuthenticationError: Error code: 401 - {'error': {'message': 'Incorrect API key provided: NULL. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}.

python example_research.py wants my Github token. I'm not comfortable trusting this code to give it that yet.

So this still seems tied into OpenAPI pretty hard.

[LanceLake]

I've dived into this a bit and figured out how to place my local LLM stuff into the base code. I will do a pull request soon for it, but I need to fix one thing with misc.py..

So it calls the OpenAIWrapper.. and Open AI from llama_index.llms...

import autogen
from autogen import OpenAIWrapper
from time import sleep

from llama_index.llms import OpenAI

These seem to force the call to

INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 401 Unauthorized"

instead of where I put my LLM..

So in a lot of the files (including misc.py), I made this..

config_listLLM = [
    {
        "model": os.environ["LOCAL_LLM_MODEL"],
        "api_key": os.environ["LOCAL_LLM_API_KEY"],
        "base_url": os.environ["LOCAL_LLM_URL"],
    }
]

and I went through and replaced config_3 and config_4 with config_listLLM.

This works fine until the reply back to the user.

INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 401 Unauthorized"
Traceback (most recent call last):
  File "E:\autogen-agi\autogen_LocalTest.py", line 78, in <module>
    user_proxy.initiate_chat(
  File "C:\Python311\Lib\site-packages\autogen\agentchat\conversable_agent.py", line 544, in initiate_chat
    self.send(self.generate_init_message(**context), recipient, silent=silent)
  File "C:\Python311\Lib\site-packages\autogen\agentchat\conversable_agent.py", line 344, in send
    recipient.receive(message, self, request_reply, silent)
  File "C:\Python311\Lib\site-packages\autogen\agentchat\conversable_agent.py", line 475, in receive
    reply = self.generate_reply(messages=self.chat_messages[sender], sender=sender)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\autogen\agentchat\conversable_agent.py", line 887, in generate_reply
    final, reply = reply_func(self, messages=messages, sender=sender, config=reply_func_tuple["config"])
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\autogen-agi\autogen_mods\modified_group_chat.py", line 450, in run_chat
    speaker = groupchat.select_speaker(speaker, self)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\autogen-agi\autogen_mods\modified_group_chat.py", line 266, in select_speaker
    extracted_next_actor = light_llm4_wrapper(
                           ^^^^^^^^^^^^^^^^^^^
  File "E:\autogen-agi\utils\misc.py", line 112, in light_llm4_wrapper
    return light_llm_wrapper(llm4, query)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\autogen-agi\utils\misc.py", line 89, in light_llm_wrapper
    response = llm.complete(query)
               ^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\llama_index\llms\base.py", line 313, in wrapped_llm_predict
    f_return_val = f(_self, *args, **kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\llama_index\llms\openai.py", line 218, in complete
    return complete_fn(prompt, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\llama_index\llms\generic_utils.py", line 153, in wrapper
    chat_response = func(messages, **kwargs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\llama_index\llms\openai.py", line 254, in _chat
    response = self._client.chat.completions.create(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\openai\_utils\_utils.py", line 299, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\openai\resources\chat\completions.py", line 556, in create
    return self._post(
           ^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\openai\_base_client.py", line 1055, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\openai\_base_client.py", line 834, in request
    return self._request(
           ^^^^^^^^^^^^^^
  File "C:\Python311\Lib\site-packages\openai\_base_client.py", line 877, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.AuthenticationError: Error code: 401 - {'error': {'message': 'Incorrect API key provided: NULL. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}

So there is some hardcoding of the URL (which makes sense).

I'm not sure how to fix it though as it seems something in a package and my skill level isn't good enough to be editing the packages directly.

Can there be a flag in the package that if a LLM is defined in .env, then it can use that url instead of the openapi one?

[JKHeadley]

Thanks @LanceLake and @dlaliberte for looking into this. I'll try to make local LLM compatibility a priority when I get some more time to work on the project.

[kripper]