Failed to communicate with the API!

[silverbladeg]

Hi, I have the ooba server running on a separate machine, i can access the chat ui and the api flag is turned on. I was able to get the integration from this project installed and configured in home assistant and it found the server and the model correctly. However, when i go to the Assist box and type in a query, it stalls out for a bit before finally giving me "Failed to communicate with the API!" message in Assist.

Initially i saw a couple error messages show up on the server, now nothing shows up on the ooba server. the errors were: 1) Could not find the character "[]" inside characters/. No character has been loaded. on another attempt i saw: 2) Llama.generate: prefix-match hit

Any help would be much appreciated. Thanks

[vhsdream]

Try restarting the textgen webui with the --verbose flag so you can see what else might be happening. I can confirm that the character tab under 'Parameters' is bugged for me as well, but the Assistant still works for me in HA. 'Prefix-match hit' indicates it was working correctly, at least for a moment.

[acon96]

I just pushed a large update (v0.2) can you re-try with the new version? The update had a lot of bug fixes in it. If this still isn't working then please enable debug logging for the component and post the logs back here.

[neowisard]

I use ooba's text (2 GPU , 2 CPU), with API in the style of OpenAI API, for a chatbot in Telegram which caches requests and context in Mongo, code here: https://github.com/father-bot/chatgpt_telegram_bot/tree/main/bot, and if you can support OpenAI API with context caching for at least 90 minutes for instruct models - it could be a coherent scenario and a super solution that can be connected to everything. I am a beginner in home assistant and not familiar with Python, but as a DevOps and tester, I can test any integrations. I will install your code and test it for a couple of days, let you know.

[neowisard]

Failed to communicate with the API! 401 Client Error: Unauthorized for url: http://192.168.0.55:5000/v1/completions Have this error because need openai_api_key = "openai_api_key", how can i add this? for generic OpenAI API compatible interface

[acon96]

This should be fixed in v0.2.1

[neowisard]

Good afternoon, unfortunately the problem has not been solved. i am use external instance ooba with api for another projects, ooba's api (generic Open AI) works perfectly with mixtral instruct models. The thing is that for generic OpenAI API - token is not a basic auth, it is a header field Bearer, as https://github.com/oobabooga/text-generation-webui/pull/3334/commits/aea447c0a8e0dc588f7752f3db4c1e04e25c548b. In spite of this I also retested version 0.2.0 with token removed in api text-generation-webui (and 0.2.1 with token specified) and also in debugging mode of home assistant. Unfortunately I can't find the error, maybe you can tell me where to find the detailed exeptions of the llama conversation application itself ?

[silverbladeg]

hello @acon96 , apologies I was traveling and couldn't try the fix. I upgraded everything and reinstalled ooba. unforuntately, now in ooba, i get teh following assertion error while loading the home-llm model(s) Traceback (most recent call last):

File "/root/containers/text-generation-webui/modules/ui_model_menu.py", line 213, in load_model_wrapper

shared.model, shared.tokenizer = load_model(selected_model, loader)

                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/root/containers/text-generation-webui/modules/models.py", line 87, in load_model

output = load_func_maploader

     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/root/containers/text-generation-webui/modules/models.py", line 250, in llamacpp_loader

model, tokenizer = LlamaCppModel.from_pretrained(model_file)

               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/root/containers/text-generation-webui/modules/llamacpp_model.py", line 101, in from_pretrained

result.model = Llama(**params)

           ^^^^^^^^^^^^^^^

File "/root/containers/text-generation-webui/installer_files/env/lib/python3.11/site-packages/llama_cpp/llama.py", line 962, in init

self._n_vocab = self.n_vocab()

            ^^^^^^^^^^^^^^

File "/root/containers/text-generation-webui/installer_files/env/lib/python3.11/site-packages/llama_cpp/llama.py", line 2274, in n_vocab

return self._model.n_vocab()

   ^^^^^^^^^^^^^^^^^^^^^

File "/root/containers/text-generation-webui/installer_files/env/lib/python3.11/site-packages/llama_cpp/llama.py", line 251, in n_vocab

assert self.model is not None

   ^^^^^^^^^^^^^^^^^^^^^^

AssertionError