bind_tools NotImplementedError when using ChatOllama

[hyhzl]

Checked other resources

• [X] I added a very descriptive title to this issue.
• [X] I searched the LangChain documentation with the integrated search.
• [X] I used the GitHub search to find a similar question and didn't find it.
• [X] I am sure that this is a bug in LangChain rather than my code.
• [X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

def init_ollama(model_name:str = global_model):

llm = Ollama(model=model_name)

llm = ChatOllama(model=model_name)
return llm

llm = init_ollama() llama2 = init_ollama(model_name=fallbacks) llm_with_fallbacks = llm.with_fallbacks([llama2])

def agent_search(): search = get_Tavily_Search() retriver = get_milvus_vector_retriver(get_webLoader_docs("https://docs.smith.langchain.com/overview"),global_model) retriver_tool = create_retriever_tool( retriver, "langsmith_search", "Search for information about LangSmith. For any questions about LangSmith, you must use this tool!", ) tools = [search,retriver_tool]

llm = ChatOpenAI(model="gpt-3.5-turbo-0125", temperature=0) # money required

prompt = hub.pull("hwchase17/openai-functions-agent")
agent = create_tool_calling_agent(llm,tools,prompt) # no work.   
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)
agent_executor.invoke({"input": "hi!"})

Error Message and Stack Trace (if applicable)

Traceback (most recent call last): File "agent.py", line 72, in agent = create_tool_calling_agent(llm,tools,prompt) File "/home/anaconda3/envs/languagechain/lib/python3.8/site-packages/langchain/agents/tool_calling_agent/base.py", line 88, in create_tool_calling_agent llm_with_tools = llm.bind_tools(tools) File "/home/anaconda3/envs/languagechain/lib/python3.8/site-packages/langchain_core/language_models/chat_models.py", line 912, in bind_tools raise NotImplementedError() NotImplementedError

Description

because ollama provide great convenient for developers to develop and practice LLM app, so hoping this issue to be handled as soon as possible Appreciate sincerely !

System Info

langchain==0.1.19 platform: centos python version 3.8.19

[sbusso]

@hyhzl, no random mention, please.

[subhash137]

Any one gt the solution for that

[subhash137]

Even structuredoutut is not working

error -

[tcztzy]

You can use Ollama's OpenAI compatible API like

from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
    api_key="ollama",
    model="llama3",
    base_url="http://localhost:11434/v1",
)
llm = llm.bind_tools(tools)

Pretend the Ollama mode as OpenAI and have fun with the LLM developping!

[subhash137]

Thank you for your reply

On Sat, 11 May 2024, 13:12 Tang Ziya, @.***> wrote:

You can use Ollama's OpenAI compatible API https://ollama.com/blog/openai-compatibility like

from langchain_openai import ChatOpenAIllm = ChatOpenAI( api_key="ollama", model="llama3", base_url="http://localhost:11434/v1", )llm = llm.bind_tools(tools)

Pretend the Ollama mode as OpenAI and have fun with the LLM developping!

— Reply to this email directly, view it on GitHub https://github.com/langchain-ai/langchain/issues/21479#issuecomment-2105618237, or unsubscribe https://github.com/notifications/unsubscribe-auth/AW6YNHNN3T6STIR6I6LHJSTZBXDVBAVCNFSM6AAAAABHOU3E4KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBVGYYTQMRTG4 . You are receiving this because you commented.Message ID: @.***>

[alexanderp99]

@subhash137 . According to the Ollama docs, their Chat Completions API does not support function calling yet. Did you have any success?

[subhash137]

Yes, I did.

On Mon, 13 May 2024, 19:24 Alexander, @.***> wrote:

@subhash137 https://github.com/subhash137 . According to the Ollama docs https://github.com/ollama/ollama/blob/main/docs/openai.md, their Chat Completions API does not support function calling yet. Did you have any success?

— Reply to this email directly, view it on GitHub https://github.com/langchain-ai/langchain/issues/21479#issuecomment-2107641775, or unsubscribe https://github.com/notifications/unsubscribe-auth/AW6YNHLMHTGSQSIIM7XMBDDZCDAYTAVCNFSM6AAAAABHOU3E4KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBXGY2DCNZXGU . You are receiving this because you were mentioned.Message ID: @.***>

[alexanderp99]

@subhash137 would you please show, how you achieved function calling in that way?

[kaminwong]

@subhash137 would you please show, how you achieved function calling in that way?

tcztzy's comment should work

[subhash137]

from langchain_openai import ChatOpenAI llm = ChatOpenAI( api_key="ollama", model="llama3", base_url="http://localhost:11434/v1", ) llm = llm.bind_tools(tools)

If you want to run locally use LM studio download the models , run the server give the api to base_url. but i prefer to use groq for faster and efficient output.

On Tue, 14 May 2024 at 13:17, KM @.***> wrote:

@subhash137 https://github.com/subhash137 would you please show, how you achieved function calling in that way?

tcztzy's comment should work

— Reply to this email directly, view it on GitHub https://github.com/langchain-ai/langchain/issues/21479#issuecomment-2109506407, or unsubscribe https://github.com/notifications/unsubscribe-auth/AW6YNHNVDN2RXAOE2HL4Z3LZCG6PTAVCNFSM6AAAAABHOU3E4KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBZGUYDMNBQG4 . You are receiving this because you were mentioned.Message ID: @.***>

[kaminwong]

@subhash137 would you please show, how you achieved function calling in that way?

Oh sorry I just tried, seems the tools are not invoked this way. Did someone successfully make the model use the tools provided?

[subhash137]

Oh 😳 , I am sorry I didn't implemented it correctly. Just now I ran the code it is running successfully but tools are not invoked. Aa.. I don't have any idea now

On Tue, 14 May 2024, 16:21 KM, @.***> wrote:

@subhash137 https://github.com/subhash137 would you please show, how you achieved function calling in that way?

Oh sorry I just tried, seems the tools are not invoked this way. Did someone successfully make the model use the tools provided?

— Reply to this email directly, view it on GitHub https://github.com/langchain-ai/langchain/issues/21479#issuecomment-2109888211, or unsubscribe https://github.com/notifications/unsubscribe-auth/AW6YNHNRKDUWHP6VCW45263ZCHUCZAVCNFSM6AAAAABHOU3E4KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBZHA4DQMRRGE . You are receiving this because you were mentioned.Message ID: @.***>

[AmirMohamadBabaee]

I faced this error too. is there any quick fix for this problem? using OllamaFunctions can fix it?

[lalanikarim]

20881 (merged) already added bind_tools feature into OllamaFunctions

21625 (pending merge) adds support for tool_calls

[Harsh-Kesharwani]

@lalanikarim can i use chat model along with function calling. As i see chatOllama does not supports bind_tools, but in documentation it is given how to bind_tools with chatOllama.

[ErfanMomeniii]

We should use OllamaFunctions and pass the LLaMA model name as a parameter, as it includes a suitable bind_tools method for adding tools to the chain. The ChatOllama class does not possess any methods for this purpose. for more details see https://python.langchain.com/v0.1/docs/integrations/chat/ollama_functions Alternatively, we can manage this manually by defining new classes that inherit from ChatOllama, incorporating tools as parameters, and creating an appropriate invoke function to utilize these tools. @Harsh-Kesharwani

[lalanikarim]

@lalanikarim can i use chat model along with function calling. As i see chatOllama does not supports bind_tools, but in documentation it is given how to bind_tools with chatOllama.

@Harsh-Kesharwani Like @ErfanMomeniii suggested, you can use OllamaFunctions if you need function calling capabilities with Ollama. OllamaFunctions inherits from ChatOllama and adds newer bind_tools and with_structured_output functions as well as adds tool_calls property to AIMessage. While you can currently already use OllamaFunctions for function calling, there is an unmerged PR #21625 that fixes the issue where you want a chat response from OllamaFunctions in case none of the provided functions are appropriate for the request. I am hoping that to be merged sometime this week.

[ntelo007]

Can someone please post a mini example of tool calling with these pr merges?

[KIC]

I am still not able to get it to work:

from langchain_community.chat_models import ChatOllama
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_experimental.llms.ollama_functions import OllamaFunctions
from langchain_core.pydantic_v1 import BaseModel, Field
from langchain.agents import AgentExecutor, create_tool_calling_agent, tool

@tool
def magic_function(input: int):
    """applies magic function to an input"""
    return input * -2

prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "You are a helpful assistant"),
        ("human", "{input}"),
        MessagesPlaceholder("agent_scratchpad")
    ]
)

tools = [magic_function]

model = OllamaFunctions(
    model="llama3",
    # formal="json",    # commented or not, does not change the error
    keep_alive=-1,
    temperature=0,
    max_new_tokes=512,
)

agent = create_tool_calling_agent(model, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

agent_executor.invoke({"input":"What is the value of magic_function(3)"})

TypeError: Object of type StructuredTool is not JSON serializable

[lalanikarim]

I am still not able to get it to work:

from langchain_community.chat_models import ChatOllama
from langchain_core.output_parsers import StrOutputParser
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
from langchain_experimental.llms.ollama_functions import OllamaFunctions
from langchain_core.pydantic_v1 import BaseModel, Field
from langchain.agents import AgentExecutor, create_tool_calling_agent, tool

@tool
def magic_function(input: int):
    """applies magic function to an input"""
    return input * -2

prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "You are a helpful assistant"),
        ("human", "{input}"),
        MessagesPlaceholder("agent_scratchpad")
    ]
)

tools = [magic_function]

model = OllamaFunctions(
    model="llama3",
    # formal="json",    # commented or not, does not change the error
    keep_alive=-1,
    temperature=0,
    max_new_tokes=512,
)

agent = create_tool_calling_agent(model, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

agent_executor.invoke({"input":"What is the value of magic_function(3)"})

TypeError: Object of type StructuredTool is not JSON serializable

This PR fixed the JSON serialization error and couple other things.

22339

[lalanikarim]

Example notebook with tool calling from withing LangGraph agent. https://github.com/lalanikarim/notebooks/blob/main/LangGraph-MessageGraph-OllamaFunctions.ipynb

Since #22339 is not yet merged, the notebook installs langchain-expermental from my repo (source for #22339).

[Harsh-Kesharwani]

@lalanikarim does agent carries context that is return by tool at every iteration, suppose i have 3 tools below is the execution flow:

agent... use tool 1 tool 1 response: resp1

agent... (does agent carries resp1 or summarization or knowledge graph for it) use tool 2 tool 2 response: resp2

agent... (does agent carries resp1, resp2 or summarization or knowledge graph for it) use tool 3 tool 3 response: resp3

The question is does agent carries tool response as a context for next iteration.

[lalanikarim]

@lalanikarim does agent carries context that is return by tool at every iteration, suppose i have 3 tools below is the execution flow:

agent... use tool 1 tool 1 response: resp1

agent... (does agent carries resp1 or summarization or knowledge graph for it) use tool 2 tool 2 response: resp2

agent... (does agent carries resp1, resp2 or summarization or knowledge graph for it) use tool 3 tool 3 response: resp3

The question is does agent carries tool response as a context for next iteration.

@Harsh-Kesharwani

You provide an initial state on every irritation. Unless you pass the previous context into the next iteration, the agent starts with a fresh state every time. I hope this answers your question.

initial_state = ...
updated_state = agent.invoke(initial_state)

next_initial_state = <combine updated_state and a new initial state>
updated_state = agent.invoke(next_initial_state)

[Harsh-Kesharwani]

@lalanikarim can i log the prompt which is passed to the agent.

[lalanikarim]

@lalanikarim can i log the prompt which is passed to the agent.

@Harsh-Kesharwani I have included langtrace links for multiple runs in the notebook. Take a look and let me know if that answers your questions.

[lalanikarim]

Take a look at #22339 which should have addressed this issue. The PR was approved and merged yesterday but a release is yet to be cut from it and should happen in the next few days.

In the meantime, you may try and install langchain-experimental directly from langchain's source like this:

pip install git+https://github.com/langchain-ai/langchain.git\#egg=langchain-experimental\&subdirectory=libs/experimental

I hope this helps.

[Vishnullm]

@lalanikarim

I have been following your work to make tool calling happen for the OLLAMA.

https://github.com/langchain-ai/langchain/blob/master/libs/experimental/langchain_experimental/llms/ollama_functions.py

I have used the sorce code mentioned in the link for the OllamaFunctions. Eventhough I am able to convert the tools, I am still getting the error "Error executing agent: Object of type QuerySQLDataBaseTool is not JSON serializable ". The mistake maybe from my side but I am unable to figure it out. Below is the error for my code:

**Successfully converted tool: {'name': 'sql_db_query', 'parameters': {'title': '_QuerySQLDataBaseToolInput', 'type': 'object', 'properties': {'query': {'title': 'Query', 'description': 'A detailed and correct SQL query.', 'type': 'string'}}, 'required': ['query']}, 'description': "Input to this tool is a detailed and correct SQL query, output is a result from the database. If the query is not correct, an error message will be returned. If an error is returned, rewrite the query, check the query, and try again. If you encounter an issue with Unknown column 'xxxx' in 'field list', use sql_db_schema to query the correct table fields."}

Successfully bound tools to LLM Successfully created agent Successfully created agent executor

Entering new AgentExecutor chain... Error executing agent: Object of type QuerySQLDataBaseTool is not JSON serializable**

My code: from langchain_experimental.llms.ollama_functions import OllamaFunctions, convert_to_ollama_tool from langchain.agents import Tool, create_tool_calling_agent, AgentExecutor

llm = OllamaFunctions(model = "llama3", format = "json", temperature = 0, keep_alive=-1) toolkit = SQLDatabaseToolkit(db=db, llm=llm, use_query_checker=True) tools = toolkit.get_tools()

converted_tools = [] for tool in tools: try: converted_tool = convert_to_ollama_tool(tool) print(f"Successfully converted tool: {converted_tool}") converted_tools.append(converted_tool) except Exception as e: print(f"Error converting tool: {tool}, Error: {e}")

try: llm_with_tools = llm.bind_tools(converted_tools) print("Successfully bound tools to LLM") except Exception as e: print(f"Error binding tools to LLM: {e}")

SQL_PREFIX = """You are an agent designed to interact with a SQL database. Given an input question, create a syntactically correct SQLite query to run, then look at the results of the query and return the answer. Unless the user specifies a specific number of examples they wish to obtain, always limit your query to at most 5 results. You can order the results by a relevant column to return the most interesting examples in the database. Never query for all the columns from a specific table, only ask for the relevant columns given the question. You have access to tools for interacting with the database. Only use the below tools. Only use the information returned by the below tools to construct your final answer. You MUST double check your query before executing it. If you get an error while executing a query, rewrite the query and try again.

DO NOT make any DML statements (INSERT, UPDATE, DELETE, DROP etc.) to the database.

To start you should ALWAYS look at the tables in the database to see what you can query. Do NOT skip this step. Then you should query the schema of the most relevant tables. You have access to the following tools:"""

prompt = ChatPromptTemplate.from_messages( [ ("system", SQL_PREFIX), ("human", "{input}"), ("placeholder", "{agent_scratchpad}"), ] )

try: agent = create_tool_calling_agent(llm=llm_with_tools, tools=tools, prompt=prompt) print("Successfully created agent") except Exception as e: print(f"Error creating agent: {e}")

print("Tools passed to AgentExecutor:", converted_tools)

try: agent_executor = AgentExecutor.from_agent_and_tools(agent=agent, tools=tools, verbose=True) print("Successfully created agent executor") except Exception as e: print(f"Error creating agent executor: {e}")

question = "give me only the attachment attempts for the left teats where the cow id with highest value "

try: response = agent_executor.invoke({"input": question}) print("Successfully executed agent") print(response) except Exception as e: print(f"Error executing agent: {e}")

[lalanikarim]

@lalanikarim

I have been following your work to make tool calling happen for the OLLAMA.

https://github.com/langchain-ai/langchain/blob/master/libs/experimental/langchain_experimental/llms/ollama_functions.py

I have used the sorce code mentioned in the link for the OllamaFunctions. Eventhough I am able to convert the tools, I am still getting the error "Error executing agent: Object of type QuerySQLDataBaseTool is not JSON serializable ". The mistake maybe from my side but I am unable to figure it out. Below is the error for my code:

**Successfully converted tool: {'name': 'sql_db_query', 'parameters': {'title': '_QuerySQLDataBaseToolInput', 'type': 'object', 'properties': {'query': {'title': 'Query', 'description': 'A detailed and correct SQL query.', 'type': 'string'}}, 'required': ['query']}, 'description': "Input to this tool is a detailed and correct SQL query, output is a result from the database. If the query is not correct, an error message will be returned. If an error is returned, rewrite the query, check the query, and try again. If you encounter an issue with Unknown column 'xxxx' in 'field list', use sql_db_schema to query the correct table fields."}

Successfully bound tools to LLM Successfully created agent Successfully created agent executor

Entering new AgentExecutor chain... Error executing agent: Object of type QuerySQLDataBaseTool is not JSON serializable**

My code: from langchain_experimental.llms.ollama_functions import OllamaFunctions, convert_to_ollama_tool from langchain.agents import Tool, create_tool_calling_agent, AgentExecutor

llm = OllamaFunctions(model = "llama3", format = "json", temperature = 0, keep_alive=-1) toolkit = SQLDatabaseToolkit(db=db, llm=llm, use_query_checker=True) tools = toolkit.get_tools()

converted_tools = [] for tool in tools: try: converted_tool = convert_to_ollama_tool(tool) print(f"Successfully converted tool: {converted_tool}") converted_tools.append(converted_tool) except Exception as e: print(f"Error converting tool: {tool}, Error: {e}")

try: llm_with_tools = llm.bind_tools(converted_tools) print("Successfully bound tools to LLM") except Exception as e: print(f"Error binding tools to LLM: {e}")

SQL_PREFIX = """You are an agent designed to interact with a SQL database. Given an input question, create a syntactically correct SQLite query to run, then look at the results of the query and return the answer. Unless the user specifies a specific number of examples they wish to obtain, always limit your query to at most 5 results. You can order the results by a relevant column to return the most interesting examples in the database. Never query for all the columns from a specific table, only ask for the relevant columns given the question. You have access to tools for interacting with the database. Only use the below tools. Only use the information returned by the below tools to construct your final answer. You MUST double check your query before executing it. If you get an error while executing a query, rewrite the query and try again.

DO NOT make any DML statements (INSERT, UPDATE, DELETE, DROP etc.) to the database.

To start you should ALWAYS look at the tables in the database to see what you can query. Do NOT skip this step. Then you should query the schema of the most relevant tables. You have access to the following tools:"""

prompt = ChatPromptTemplate.from_messages( [ ("system", SQL_PREFIX), ("human", "{input}"), ("placeholder", "{agent_scratchpad}"), ] )

try: agent = create_tool_calling_agent(llm=llm_with_tools, tools=tools, prompt=prompt) print("Successfully created agent") except Exception as e: print(f"Error creating agent: {e}")

print("Tools passed to AgentExecutor:", converted_tools)

try: agent_executor = AgentExecutor.from_agent_and_tools(agent=agent, tools=tools, verbose=True) print("Successfully created agent executor") except Exception as e: print(f"Error creating agent executor: {e}")

question = "give me only the attachment attempts for the left teats where the cow id with highest value "

try: response = agent_executor.invoke({"input": question}) print("Successfully executed agent") print(response) except Exception as e: print(f"Error executing agent: {e}")

@Vishnullm I'll try this later today and provide an update.

[lalanikarim]

@Vishnullm The issue is because create_tool_calling_agent rebinds the tools to the llm and doesn't give you the opportunity to convert_to_ollama_tool.

I am investigating a more permanent fix, but for now, make the following adjustments to your code:

Original

agent = create_tool_calling_agent(llm=llm_with_tools, tools=tools, prompt=prompt)

New

from langchain.agents.format_scratchpad.openai_tools import (
    format_to_openai_tool_messages,
)
from langchain.agents.output_parsers.openai_tools import OpenAIToolsAgentOutputParser

agent = (
    {
        "input": lambda x: x["input"],
        "agent_scratchpad": lambda x: format_to_openai_tool_messages(
            x["intermediate_steps"]
        ),
    }
    | prompt
    | llm_with_tools
    | OpenAIToolsAgentOutputParser()
)

This code is from the custom codes notebook

[Vishnullm]

@lalanikarim , Thanks for the suggestion. My requirement is to use mainly llama llms from Ollama. With the suggestion you provided I can only use gpt models. Is there any other way around?

I tried create_sql_agent with zeroshotpromptreactdescription. When I give any complex question which requires multiple sql tables to join or to look for multiple columuns, the agent just fails and goes in a loop of redoing the same action execution even though it sometime generate the answer it does not know that it is the correct answer.

As a result, I thought create_tool_calling_agent would be more effective for this use case, but I am facing the issue with function calling when I used OllamFunctions.

llm = OllamaFunctions(model = "llama3", format = "json", temperature = 0, keep_alive=-1)
toolkit = SQLDatabaseToolkit(db=db, llm=llm, use_query_checker=True)
tools = toolkit.get_tools()

converted_tools = [convert_to_ollama_tool(tool) for tool in tools]
llm_with_tools = llm.bind_tools(tools=converted_tools)
agent = create_tool_calling_agent(
    llm=llm_with_tools,
    tools=converted_tools,  
    prompt=prompt
)

agent_executor = AgentExecutor.from_agent_and_tools(agent=agent, tools=tools, verbose=True)

when I use this code by understanding your work on OllamaFunctions wrapper, i am getting the error: Ollama call failed with status code 400. Details: {"error":"invalid options: functions"}

I believe with this structure atleast .bind_tools() is working but not the tool_calling.

Please guide me on how to make a model suitable for sql Q/A using Ollama or local llms.

[lalanikarim]

@Vishnullm I am looking into getting OllamaFunctions working correctly with tool calling. This might take some time. I have created a notebook with a simple zero shot LangGraph agent that you might want to check out in the mean time. https://github.com/lalanikarim/notebooks/blob/main/SQL-LangGraph-Agent/SqlAgent-LangGraph.ipynb

It can currently answer simple questions but can be adapted to answer more complex questions by applying techniques like ReAct etc.

I hope this helps.

[lalanikarim]

@Vishnullm After reviewing the underlying the code for Agent and AgentExecutor, I've concluded that more work is needed on OllamaFunctions, such as stream support, before it can be used within AgentExecutor. Until that work is done, unfortunately, you won't be able to use AgentExecutor based agents. LangGraph based agents do work however, and you should be able to modify existing LangGraph agents or build new ones that use OllamaFunctions.

[Vishnullm]

Hi @lalanikarim,

my use case is to specifically use open source llms for interacting with yaml files which have the same format of key value pairs across all the files with constant key names. so i thought if i convert all these files to sql tables then can have more functionality for the chatbot that interacts with sql database. when I created the create_sql_agent with zeroshotreactdesription agent type, the model is not working really well posing me with errors as shuch shown in the below picture

As I am using llama3 which I see as a good open source llm so far, I am not able to use tool calling or open ai function agent types which are termed as best agent types online.

1) as per your understanding, can you tell which agent and agenttype is best suitable for sql chatbots and y they are best? 2) If you have any process in mind that is best suitable for the sql chat model, please share your thoughts.

your expert guidance is really appreciable. thanks for the support.

[KadriMufti]

Are there any updates in this space? I am also using Ollama and I have tried the various suggestions here, including the first one of using ChatOpenAI, and I get errors. Thank you for your efforts @lalanikarim

[lalanikarim]

Hi @lalanikarim,

my use case is to specifically use open source llms for interacting with yaml files which have the same format of key value pairs across all the files with constant key names. so i thought if i convert all these files to sql tables then can have more functionality for the chatbot that interacts with sql database. when I created the create_sql_agent with zeroshotreactdesription agent type, the model is not working really well posing me with errors as shuch shown in the below picture

As I am using llama3 which I see as a good open source llm so far, I am not able to use tool calling or open ai function agent types which are termed as best agent types online.

1. as per your understanding, can you tell which agent and agenttype is best suitable for sql chatbots and y they are best?

2. If you have any process in mind that is best suitable for the sql chat model, please share your thoughts.

your expert guidance is really appreciable. thanks for the support.

@Vishnullm @KadriMufti I have not yet tried local llms for SQL generation within agents. Having said that, couple of things I do want to mention:

1. The AgentExecutor based agents are being deprecated in favor of LangGraph (https://python.langchain.com/v0.2/docs/how_to/agent_executor/)
2. You may want to try llms fine tuned for coding tasks, like codestral etc as they may perform better on SQL tasks.
3. You may also want to investigate llms fine tuned for SQL tasks for your use case.

OllamaFunctions only supports LangGraph agents. AgentExecutor based agents are not yet supported. And given that LangChain team is deprecating them, I suggest looking into SQL agents built with LangGraph.

Here is yet to be published documentation for OllamaFunctions which also includes instructions for building LangGraph agents. https://langchain-git-fork-lalanikarim-ollama-function-fbb983-langchain.vercel.app/v0.2/docs/integrations/chat/ollama_functions/

I hope it helps.

[RyanKung]

I finally solved this issue by using litellm

ollama -> litellm proxy -> langchain's ChatOpenAI

[msssouza]

I finally solved this issue by using litellm

ollama -> litellm proxy -> langchain's ChatOpenAI

Hi. Can you share your code?

[RyanKung]

I finally solved this issue by using litellm ollama -> litellm proxy -> langchain's ChatOpenAI

Hi. Can you share your code?

1. Launch litellm with config file like:

model_list:
  - model_name: llama3
    litellm_params:
      model: ollama/llama3
      api_base: http://127.0.0.1:11434
      api_key: "ollama"
      rpm: 600

  - model_name: llama2-uncensored-70b
    litellm_params:
      model: ollama/llama2-uncensored-70b
      api_base: http://127.0.0.1:11434
      api_key: "ollama"
      rpm: 600

docker run -d --restart always --network host -v /home/ryan/lab/litellm/litellm_config.yaml:/app/config.yaml ghcr.io/berriai/litellm:main-latest --config /app/config.yaml --detailed_debug

2. Init ChatOpenAI with litellm's endpoint

llm = ChatOpenAI(api_key="ollama", base_url='http://127.0.0.1:4000/v1', model="llama3")

[humpheryxiao]

Hi @Vishnullm have you found a way out, I met the same problem when to use create_sql_agent

[anirbanbasu]

Given that Ollama now supports tool calling as of July 25, 2024 and that LangChain now suggests that we use ChatOllama instead of OllamaFunctions, I had code that was working with OllamaFunctions but failed with ChatOllama.

To be specific, changing OllamaFunctions to ChatOllama on line 69 here and loading model 'llama3-groq-tool-use caused it to fail with:

<virtualenv-path>/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 1055, in bind_tools
    raise NotImplementedError()

on line 106 here.

Needless to say that my ollama server has been updated, as has been the corresponding langchain_community package.

My take on this is that others, in this thread, have been importing the wrong ChatOllama by using from langchain_community.chat_models import ChatOllama. That is exactly what I did to get the above error. However, reading the documentation at https://python.langchain.com/v0.2/docs/integrations/chat/ollama/, we ought to install the langchain-ollama package and import ChatOllama as from langchain_ollama import ChatOllama. With that, I no longer have the error.

Does this solve the problems discussed in this thread?

[lalanikarim]

importing ChatOllama from langchain_ollama does solve the original issue of bind_tools NotImplementedError. It is important to note however, that the new ChatOllama still doesn't implement .with_structured_output(). So OllamaFunctions still has some use until that is taken care of.

Given that Ollama now supports tool calling as of July 25, 2024 and that LangChain now suggests that we use ChatOllama instead of OllamaFunctions, I had code that was working with OllamaFunctions but failed with ChatOllama.

To be specific, changing OllamaFunctions to ChatOllama on line 69 here and loading model 'llama3-groq-tool-use caused it to fail with:

<virtualenv-path>/lib/python3.12/site-packages/langchain_core/language_models/chat_models.py", line 1055, in bind_tools
    raise NotImplementedError()

on line 106 here.

Needless to say that my ollama server has been updated, as has been the corresponding langchain_community package.

My take on this is that others, in this thread, have been importing the wrong ChatOllama by using from langchain_community.chat_models import ChatOllama. That is exactly what I did to get the above error. However, reading the documentation at https://python.langchain.com/v0.2/docs/integrations/chat/ollama/, we ought to install the langchain-ollama package and import ChatOllama as from langchain_ollama import ChatOllama. With that, I no longer have the error.

Does this solve the problems discussed in this thread?

[whatwehaveunlearned]

Do we know why it works with the ChatOllama import and not the community one? Is the plan is to eventually make it work with the community packages as well? I am a bit confused about this.

[lalanikarim]

The new ChatOllama implementatin from the langchain_ollama package should be used for tool use and structured outputs. You'll need to use it with a model that the ollama team has tagged for tool use. Check the model page on ollama.com and see if it has "Tools" label.

[raj-acrivon]

If I run the same code with ChatGroq I am able to retrieve relevant table

What I am doing wrong, My Goal is to use it with LangGraph so I structured fake_state.

llm = ChatOllama( model="llama3-groq-tool-use:8b", temperature=0, top_k=10, top_p = 0.5, repeat_penalty = 1.3,

num_thread = multiprocessing.cpu_count() -1,

)

db = SQLDatabase.from_uri("sqlite:////home/azureuser/ai_chatbot/sample.db", sample_rows_in_table_info=3)

toolkit = SQLDatabaseToolkit(db=db, llm=llm) tools = toolkit.get_tools()

class ToolInput(BaseModel): tables: str = Field(description = "Input is a comma-separated list of relevant tables (e.g., 'table1, table2, table3')")

@tool("get_schema_tool_use", args_schema = ToolInput) def get_schema_tool_use(tables:str) -> str: """ Input is a comma-separated list of relevant tables, output is the schema and sample rows for those tables. Be sure that the tables actually exist in list_tables_tool output. Example Input tables: table1, table2, table3 """ return next(tool for tool in tools if tool.name == "sql_db_schema").invoke(tables)

from langchain_core.prompts.chat import BaseMessage, HumanMessage, AIMessage, ChatPromptValue

local llm with tool access

llm_with_tools = llm.bind_tools( tools = [get_schema_tool_use], tool_choice = {"type":"function", "function": {"name":"get_schema_tool_use"}}, )

Example fake state with a list of messages

fake_state = ChatPromptValue(messages= [HumanMessage(content='What is the mean phosphorylation level of the X across samples in the Y from quant table?', id='a6bab255-2619-4b95-90b9-f3d8b321a99a'), AIMessage(content='', id='75948a2f-0ba8-4249-bdb4-8a2e6b447bc', tool_calls=[{'name': 'list_tables_tool', 'args': {}, 'id': 'tool_abcd123', 'type': 'tool_call'}]), ToolMessage(content='de, description, gsea, metadata,, quant', name='list_tables_tool', id='4a94a0f0-8481-404c-8957-b5e8acf3d2da', tool_call_id='tool_abcd123'), ])

response = llm_with_tools.invoke(fake_state) reponse #tool_calls = [] (AIMessage(content="I'm sorry but I do not have enough information to complete that task. Can you provide more details or clarify your question?", response_metadata={'model': 'llama3-groq-tool-use:8b', 'created_at': '2024-09-11T17:05:40.432974842Z', 'message': {'role': 'assistant', 'content': "I'm sorry but I do not have enough information to complete that task. Can you provide more details or clarify your question?"}, 'done_reason': 'stop', 'done': True, 'total_duration': 26541844067, 'load_duration': 2809424482, 'prompt_eval_count': 299, 'prompt_eval_duration': 19489875000, 'eval_count': 26, 'eval_duration': 4114271000}, id='run-5a5f656d-9217-4668-97d2-d545487f0d82-0', usage_metadata={'input_tokens': 299, 'output_tokens': 26, 'total_tokens': 325}) )

If I run the same code with ChatGroq I am able to retrieve relevant table

What I am doing wrong, My Goal is to use it with LangGraph so I structured fake_state.

I am able to run belowcode, but not above code: %%time

class UserInput(BaseModel): location: str = Field(description = "The city and state of the user input e.g.: Los Angeles") unit: str = Field(..., description = "Unit of measurement for weather (e.g., 'Fahrenheit', 'CELSIUS')")

@tool("weather_monuments_call", args_schema = UserInput) def weather_monuments_call(location: str, unit: str, city_traveling_spots: list[str]) -> str: """Get weather and best three places to visit at given location""" return f"Current weather in {location} is 22 {unit}."

local llm with tool access

llm_with_tools = llm.bind_tools( tools = [weather_monuments_call], tool_choice = {"type":"function", "function": {"name":"weather_monuments_call"}} )

response = llm_with_tools.invoke("What is weather in San Fracisco in celsius and what are best places to visit in San Fracisco?") response.tool_calls

I have exhausted a lot of available online blogs but can't resolve my error. Can anyone help on this :)

[lalanikarim]

Groq has a their own internal proprietary implementation for tool use that is likely different from Ollama's. In this situation, you might look into adjusting your tool definition or look into a different model. llama3.1 is supported for tool use natively by Ollama and might yield better outcome.

If I run the same code with ChatGroq I am able to retrieve relevant table

What I am doing wrong, My Goal is to use it with LangGraph so I structured fake_state.

llm = ChatOllama( model="llama3-groq-tool-use:8b", temperature=0, top_k=10, top_p = 0.5, repeat_penalty = 1.3, # num_thread = multiprocessing.cpu_count() -1, )

db = SQLDatabase.from_uri("sqlite:////home/azureuser/ai_chatbot/sample.db", sample_rows_in_table_info=3)

toolkit = SQLDatabaseToolkit(db=db, llm=llm) tools = toolkit.get_tools()

class ToolInput(BaseModel): tables: str = Field(description = "Input is a comma-separated list of relevant tables (e.g., 'table1, table2, table3')")

@tool("get_schema_tool_use", args_schema = ToolInput) def get_schema_tool_use(tables:str) -> str: """ Input is a comma-separated list of relevant tables, output is the schema and sample rows for those tables. Be sure that the tables actually exist in list_tables_tool output. Example Input tables: table1, table2, table3 """ return next(tool for tool in tools if tool.name == "sql_db_schema").invoke(tables)

from langchain_core.prompts.chat import BaseMessage, HumanMessage, AIMessage, ChatPromptValue

local llm with tool access

llm_with_tools = llm.bind_tools( tools = [get_schema_tool_use], tool_choice = {"type":"function", "function": {"name":"get_schema_tool_use"}}, )

Example fake state with a list of messages

fake_state = ChatPromptValue(messages= [HumanMessage(content='What is the mean phosphorylation level of the X across samples in the Y from quant table?', id='a6bab255-2619-4b95-90b9-f3d8b321a99a'), AIMessage(content='', id='75948a2f-0ba8-4249-bdb4-8a2e6b447bc', tool_calls=[{'name': 'list_tables_tool', 'args': {}, 'id': 'tool_abcd123', 'type': 'tool_call'}]), ToolMessage(content='de, description, gsea, metadata,, quant', name='list_tables_tool', id='4a94a0f0-8481-404c-8957-b5e8acf3d2da', tool_call_id='tool_abcd123'), ])

response = llm_with_tools.invoke(fake_state) reponse #tool_calls = [] (AIMessage(content="I'm sorry but I do not have enough information to complete that task. Can you provide more details or clarify your question?", response_metadata={'model': 'llama3-groq-tool-use:8b', 'created_at': '2024-09-11T17:05:40.432974842Z', 'message': {'role': 'assistant', 'content': "I'm sorry but I do not have enough information to complete that task. Can you provide more details or clarify your question?"}, 'done_reason': 'stop', 'done': True, 'total_duration': 26541844067, 'load_duration': 2809424482, 'prompt_eval_count': 299, 'prompt_eval_duration': 19489875000, 'eval_count': 26, 'eval_duration': 4114271000}, id='run-5a5f656d-9217-4668-97d2-d545487f0d82-0', usage_metadata={'input_tokens': 299, 'output_tokens': 26, 'total_tokens': 325}) )

If I run the same code with ChatGroq I am able to retrieve relevant table

What I am doing wrong, My Goal is to use it with LangGraph so I structured fake_state.

I am able to run belowcode, but not above code: %%time

class UserInput(BaseModel): location: str = Field(description = "The city and state of the user input e.g.: Los Angeles") unit: str = Field(..., description = "Unit of measurement for weather (e.g., 'Fahrenheit', 'CELSIUS')")

@tool("weather_monuments_call", args_schema = UserInput) def weather_monuments_call(location: str, unit: str, city_traveling_spots: list[str]) -> str: """Get weather and best three places to visit at given location""" return f"Current weather in {location} is 22 {unit}."

local llm with tool access

llm_with_tools = llm.bind_tools( tools = [weather_monuments_call], tool_choice = {"type":"function", "function": {"name":"weather_monuments_call"}} )

response = llm_with_tools.invoke("What is weather in San Fracisco in celsius and what are best places to visit in San Fracisco?") response.tool_calls

I have exhausted a lot of available online blogs but can't resolve my error. Can anyone help on this :)

[Foxify52]

What's the status on this issue?

[zhouhh2017]

remote connection led the failure of bind_tools

When I locally use Ollama, it calls the tools correctly. But langchain_ollama.ChatOllama and langchain_openai.ChatOpenAI not work, while the Ollama running remotely. Since not only one API function or LLM have this problem, the error may happen in a general place. For example, llm.bind_tools cannot send the tool information to remote Ollama. Details locally use works well

# start ollama
ollama serve
ollama pull llama3.2

# run agent
from langchain_ollama import ChatOllama
llm = ChatOllama(
        model = "llama3.2", 
        temperature = 0.3,
    )
...
part_3_assistant_runnable = assistant_prompt | llm.bind_tools(
    part_3_safe_tools + part_3_sensitive_tools
)

The Agent output: =================== Ai Message ================= Tool Calls: list_selectable_catalogs Call ID: .. Args: catalogue_name: Gaia catalogue_type: 0 ================== Tool Message ================= Name: list_selectable_catalogs { "code": 200, ...

remote ollama cannot see the tools

# start remote ollama 
export OLLAMA_HOST=0.0.0.0
ollama serve

# run agent
from langchain_ollama import ChatOllama
llm = ChatOllama(
        base_url="http://X.X.X.X:11434/",
        model = "llama3.2", 
        temperature = 0.3,
    )
# or use ChatOpenAI
from langchain_openai import ChatOpenAI
llm = ChatOpenAI(
     api_key="ollama",
     base_url="http://X.X.X.X:11434/v1",
     model = "llama3.2", 
     temperature = 0.3,
 )
...no any other change

The Agent output: To assist you in finding the Gaia catalog, I will follow these steps:

1. Confirm the type of data required: Please inform me of the specific type of data you need. For instance, if you are interested in the position, distance, spectral type, or any other particular attribute of stars, please provide detailed information.
2. Query the database: Use astronomical database tools (such as the SIMBAD Astronomical Database or the Gaia Data Release) to perform the search. These tools allow you to search and retrieve catalog data based on specific criteria. ...

System Info langchain==0.3.0 langchain_community==0.3.0 langgraph==0.2.28 langchain_ollama==0.2.0 platform: Unbantu Python 3.12.4

[gnf1]

I used bind_tools to solve the original problem of NotImplementedErrs, but now I have an error of does not implement on_chat_model_start" in the langchain_core callback and it turns out that this is not implemented anywhere, is there any other package that I can copy?