[feature request] local only

[Tom-Neverwinter]

having the ability to use the api to paid services is cute and all.

can we have local only.

nobody wants to pay for these services anymore especially as llama3.1 blew them away with costly tiers and other bs.

comparison of ai agents

https://github.com/SamurAIGPT/Best-AI-Agents

[maximuslee1226]

I don't know why he listed a bunch of other models when they don't work. # chat_llm = models.get_groq_llama70b(temperature=0.2)

chat_llm = models.get_groq_llama70b_json(temperature=0.2)

# chat_llm = models.get_groq_llama8b(temperature=0.2)
# chat_llm = models.get_openai_gpt35(temperature=0)
# chat_llm = models.get_openai_gpt4o(temperature=0)
# chat_llm = models.get_openai_chat(temperature=0)
# chat_llm = models.get_anthropic_opus(temperature=0)
# chat_llm = models.get_anthropic_sonnet(temperature=0)
chat_llm = models.get_anthropic_sonnet_35(temperature=0)
# chat_llm = models.get_anthropic_haiku(temperature=0)
# chat_llm = models.get_ollama_dolphin()
# chat_llm = models.get_ollama(model_name="gemma2:27b")
# chat_llm = models.get_ollama(model_name="llama3:8b-text-fp16")
# chat_llm = models.get_ollama(model_name="llama3.1:latest")
# chat_llm = models.get_ollama(model_name="gemma2:latest")
# chat_llm = models.get_ollama(model_name="qwen:14b")
# chat_llm = models.get_google_chat()

[marcello-mix]

It keeping get error asking for API KEY even if we choose llama3... Is there a way to use it with local LLM like Ollama + LLAMA3?

[Jbollenbacher]

This is possible already. Try this configuration at the top of main.py:

def initialize():

    # main chat model used by agents (smarter, more accurate)

    # chat_llm = models.get_groq_llama70b(temperature=0.2)
    # chat_llm = models.get_groq_llama70b_json(temperature=0.2)
    # chat_llm = models.get_groq_llama8b(temperature=0.2)
    # chat_llm = models.get_openai_gpt35(temperature=0)
    # chat_llm = models.get_openai_gpt4o(temperature=0)
    # chat_llm = models.get_openai_chat(temperature=0)
    # chat_llm = models.get_anthropic_opus(temperature=0)
    # chat_llm = models.get_anthropic_sonnet(temperature=0)
    # chat_llm = models.get_anthropic_sonnet_35(temperature=0)
    # chat_llm = models.get_anthropic_haiku(temperature=0)
    # chat_llm = models.get_ollama_dolphin()
    # chat_llm = models.get_ollama(model_name="gemma2:27b")
    chat_llm = models.get_ollama(model_name="llama3.1")
    # chat_llm = models.get_ollama(model_name="gemma2:latest")
    # chat_llm = models.get_ollama(model_name="qwen:14b")
    # chat_llm = models.get_google_chat()

    # utility model used for helper functions (cheaper, faster)
    # utility_llm = models.get_openai_chat(temperature=0)
    utility_llm = models.get_ollama(model_name="llama3.1")

    # embedding model used for memory
    # embedding_llm = models.get_embedding_openai()
    embedding_llm = models.get_embedding_hf()

Though to be honest, it's not very good with llama3.1:8b out of the box. It struggles with the tool use formatting. It's possible tweaking the system prompt may help, but I havent tried too much yet.

[mgscox]

Try to be clearer in your system prompt - the below is extracted from my automatic computer operator system prompt, and L3.1 follows it.

### Response format ###
You MUST respond in RFC 8259 strictly compliant JSON format without deviation or explanation using the schema defined below. Note that you MUST return an array even if your plan only includes a single action.

[{
    "function_call": {
        "name": "name of the action to execute",
        "args": "parameters for the action as a string, number, or RFC-compliant JSON representation of an object"
    },
    "reflection": "after reviewing your historic actions, a short reflection of what aspect of the goal you have accomplished so far and a short summary of your plan to complete",
    "rationale": "the reason why you are calling the function",
    "outcome": "what change you expect to see on the user's desktop if the action is successful"
}]

#### Example response ####
[{
    "function_call": {
        "name": "mouse_move",
        "args": {"x": "100", "y": "200"}
    },
    "reflection": "i have located the icon for miscrosoft word on the user's desktop, which I will click to open enabling me to write the poem",
    "rationale": "I need the cursor to be over the icon so I can double-click it",
    "outcome": "the mouse will be over the microsoft word icon"
}, {
    "function_call": {
        "name": "click",
        "args": {"button": "left", "double": true}
    },
    "reflection": "I moved the mouse cursor over the microsoft word icon so I can double-click it which will open the application enabling me to write the poem",
    "rationale": "I need to double-click the icon to open the application",
    "outcome": "the microsoft word application will be opened and have focus for input"
}]

[247pips]

sorry can you briefly explain the process here after downloading the git hub? I’m having to use ChatGPT to edit the code to function even using the openai api key and haven’t been able to get llama 3.1 to work

[ktneely]

Is there a way to provide the agents with knowledge of a remote Ollama server, like one running on the LAN. Something like OLLAMA_BASE_URL="http://192.168.0.10:11434" as an environment variable or some declaration in main.py?

[lakshaysethi]

related : #51 and #64