yonitjio
commented
4 weeks ago From my experience, LocalAI is a bit tricky to modify.
A bit background first, the reason I forked LocalAI was because it didn't work with llama-cpp embeddings, in this case the nomic-embedding. I haven't check it again though.
The way I modified it was to replace all the codes for the llama-cpp backend with the server example from llama-cpp repo. This made it work with nomic-embed but it lost other features, I have not tried many things, but one thing for sure is that my fork removed multi-modal support.
If you still want to use my fork, I would suggest you to try to install from original repo. Make it work first, to eliminate the possibilities of missing libraries, or other requirements. After that, then you can try my fork.
If you're open to other platform, you might want to try Ollama, I have not tried it, but it seems it supports nomic embedding.
The hermes and nomic-embedding is just my naming for the model, they're not the "official" model name.
For hermes you can find it here https://github.com/yonitjio/exploring-odoo/blob/main/calendar_bot/doc/hermes.yaml.
For nomic-embedding:
name: nomic-embedding
backend: llama-cpp
mmap: true
context_size: 2048
f16: true
embeddings: true
gpu_layers: 33
parameters:
model: nomic-embed-text-v1.5.Q5_K_M.gguf
batch: 2048
usage: |
curl http://localhost:8080/v1/embeddings \
-H "Content-Type: application/json" \
-d '{ "model": "nomic-embedding", "input": "This a text" }'For adv_ai_bot, you can find the definition here https://github.com/yonitjio/exploring-odoo/blob/main/adv_ai_bot/models/agents/consts.py.
As for the Autogen, I did modify it, but I don't think it's related to the error you're experiencing. Please see https://github.com/microsoft/autogen/issues/2540 for my modification.
If you're using local AI, most of them if not all, do not need API key, well they usually need the API key to be provided but it's not used for authentication or anything so you can put any string to it.
The error you showed should be related to connection, please make sure LocalAI is running and you can access it. To test this you can use curl or something similar first, and adjust the const.py I mentioned earlier accordingly.
kalmenchia
Hi, I wish to get some advice as part of the setting, I was trying to set up your localAI fork as part of the setup. downloaded, using go1.23.1 , to build, but was not able to proceed and stuck at
it is a ubuntu 22.04 server. In AMD/64.
Can i check with you other than localAI, whether i need your forked version of AutoGen as well? For the 2 models specified in your video, it is correct if I can use the localAI ui, i.e. browser web UI to download them after localAI is installed, right? The "hermes-2" and "nomic-embed" models.
When I tried to install the adv_ai_bot, it was trying to initiate the openAI , and odoo raised an exception , executing the user_proxy.initiate_chat(analyst, message=text) , where the text is "What are the sales this week..." so do I need to set up the openAI key and other related parameters too?
I would be very grateful if u could help by providing some directions for the initial setup. Thanks.