search

langchain-ai / langchain

🦜🔗 Build context-aware reasoning applications
https://python.langchain.com
MIT License
92.04k stars 14.65k forks source link

'Bing Search Results JSON' does not match '^[a-zA-Z0-9_-]{1,64}$' - 'functions.0.name' with new langchain 0.1 when using BingSearchResults into agent #16368

Closed glejdis closed 7 months ago

glejdis commented 7 months ago

Checked other resources

Example Code

The following code ''' from langchain.agents import create_openai_tools_agent from langchain_community.utilities.bing_search import BingSearchAPIWrapper from langchain_community.tools.bing_search.tool import BingSearchRun, BingSearchResults

def create_new_bing_search_agent_function_openai(llm):

bing_search = BingSearchAPIWrapper(bing_search_url="xxx", bing_subscription_key="xxx", k=4)
bing_tool = BingSearchResults(num_results=1, api_wrapper=bing_search)

tools_bing = [bing_tool]

prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "Have a conversation with a human. You are a helpful assistant who retrieves information from the Bing Search (the internet)."),
        ("user", "{input}"),
        MessagesPlaceholder(variable_name="agent_scratchpad"),
    ]
)

agent = create_openai_functions_agent(llm, tools_bing, prompt)

agent_executor = AgentExecutor(tools=tools_bing, agent=agent)

return agent_executor

agent_chain_bing = create_new_bing_search_agent_function_openai(llm) output = agent_chain_bing.invoke({"input": "What is stock price of Apple?"}) output['output'] '''

Gives this error:

BadRequestError: Error code: 400 - {'error': {'message': "'Bing Search Results JSON' does not match '^[a-zA-Z0-9_-]{1,64}$' - 'functions.0.name'", 'type': 'invalid_request_error', 'param': None, 'code': None}}

The same error appears if I define the agent like this: agent = create_openai_tools_agent(llm, tools_bing, prompt)

Am I using this agent wrong? Is there some other way how i can use bing search in an agent?

Description

langchain==0.1.1

System Info

model GPT4 32k

Related Components

dosubot[bot] commented 7 months ago

🤖

Hey @glejdis! Good to see you back here. 😊 Thanks for providing such detailed information - it really helps expedite the troubleshooting process. I'm diving into this now, and I'll provide you with a comprehensive answer shortly. Hang tight! 🚀

jatinchawda1503 commented 7 months ago

@glejdis,

You are already using BingSearchAPIWrapper with k=4 (i.e. num of results). For more refer to the docs https://python.langchain.com/docs/integrations/tools/bing_search#number-of-results.

You can try this


llm = ChatOpenAI(temperature=0, model="gpt-3.5-turbo-0613")
search = BingSearchAPIWrapper(k=4)

tools = [
    Tool(
        name="BingSearch",
        func=search.run,
        description="useful for searching using Bing. "
    )
]
exec_agent= initialize_agent(tools, llm, agent=AgentType.OPENAI_FUNCTIONS, verbose=True)

Also, can you share the code of create_openai_functions_agent to verify what going on inside this function?

glejdis commented 7 months ago

Thank you @jatinchawda1503 ,

I was using this code, but i wanted to upgrade to the new version of langchain 0.1. initialize_agent will be deprecated. I am trying to follow this logic: https://python.langchain.com/docs/modules/agents/agent_types/openai_functions_agent

jatinchawda1503 commented 7 months ago

Thanks a lot for the clarification @glejdis,

from langchain_community.utilities import BingSearchAPIWrapper
from langchain.agents import AgentExecutor, create_openai_functions_agent

llm = ChatOpenAI(temperature=0, model="gpt-3.5-turbo-0613")
tools = [BingSearchAPIWrapper(k=4)]
agent = create_openai_functions_agent(llm, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)
agent_executor.invoke({"input": "what is LangChain?"})

This should work for you.

baskaryan commented 7 months ago

thanks for flagging @glejdis! believe #16395 should fix, openai api just doesnt support function names with spaces

glejdis commented 7 months ago

Thank you @jatinchawda1503

Your suggested code throws this error:

AttributeError: 'BingSearchAPIWrapper' object has no attribute 'args_schema'

jatinchawda1503 commented 7 months ago

@glejdis, the issue was because of the openai api just doesnt support function names with spaces in the previous version a new patch has been released (0.1.2) where they fixed this issue.

Can you try running the code after installing new version.

pip install langchain==0.1.2
glejdis commented 7 months ago

@jatinchawda1503 I did install langchain==0.1.2 but the error still persists: 'Bing Search Results JSON' does not match '^[a-zA-Z0-9_-]{1,64}$' - 'functions.0.name'

jatinchawda1503 commented 7 months ago

@glejdis, if it is possible can you share your code in colab ?

jatinchawda1503 commented 7 months ago

Hello @glejdis,

I have tested and updated the code in collab, here is the link -> https://colab.research.google.com/drive/19So2OYyrtEE5TykF-EcdECun0KP-GAq_?usp=sharing Works fine for me.

Here is the full code for your reference

from langchain_community.utilities import BingSearchAPIWrapper
from langchain.agents import AgentExecutor, create_openai_functions_agent
from langchain.prompts.chat import (
    ChatPromptTemplate,
    HumanMessagePromptTemplate,
    SystemMessagePromptTemplate,
    MessagesPlaceholder
)
from langchain.tools import StructuredTool
from langchain_community.chat_models import ChatOpenAI

search = BingSearchAPIWrapper(bing_search_url="https://api.bing.microsoft.com/v7.0/search",bing_subscription_key=bing_secret_key,k=1)
llm = ChatOpenAI(temperature=0, model="gpt-3.5-turbo-0613",openai_api_key=openai_key
prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "Have a conversation with a human. You are a helpful assistant who retrieves information from the Bing Search (the internet)."),
        ("user", "{input}"),
        MessagesPlaceholder(variable_name="agent_scratchpad"),
    ]
)

tools = [
    StructuredTool.from_function(
        name="BingSearch",
        func=search.run,
        description="useful for searching using Bing. "
    )
]

agent = create_openai_functions_agent(llm, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)
agent_executor.invoke({"input": "what is LangChain?"})

Output

> Entering new AgentExecutor chain...

Invoking: `BingSearch` with `{'query': 'LangChain'}`

<b>LangChain</b>’s flexible abstractions and extensive toolkit unlocks developers to build context-aware, reasoning LLM applications.LangChain is a platform that provides flexible abstractions and an extensive toolkit for developers to build context-aware and reasoning language learning and modeling (LLM) applications. It allows developers to create applications that can understand and reason about language in a more advanced and intelligent way.

> Finished chain.
{'input': 'what is LangChain?',
 'output': 'LangChain is a platform that provides flexible abstractions and an extensive toolkit for developers to build context-aware and reasoning language learning and modeling (LLM) applications. It allows developers to create applications that can understand and reason about language in a more advanced and intelligent way.'}
glejdis commented 7 months ago

@jatinchawda1503 thank you so so much! That worked! I have one more question. What do I need to configure to also see the intermediate_steps? I need to extract the sources and the links to the website from which the answer came from. I set intermediate_steps=True in AgentExecutor, but it did not work.

Any recommendations?

jatinchawda1503 commented 7 months ago

@glejdis, you can try metadata. BingSeachAPIWrapper gives you metadata, you can try bind this to the Agent.

I think itermediate_steps is useful when you have multiple tools.

glejdis commented 7 months ago

@jatinchawda1503 Yes you are right, running search.results("what is LangChain?", 1) gives me the sources. However I want to include that as part of the agents procedure, not as a separate step. Because then something like this happens, which I do not want to:

''' from langchain_community.utilities import BingSearchAPIWrapper from langchain.agents import AgentExecutor, create_openai_functions_agent from langchain.prompts.chat import ( ChatPromptTemplate, HumanMessagePromptTemplate, SystemMessagePromptTemplate, MessagesPlaceholder ) from langchain.tools import StructuredTool from langchain_community.chat_models import ChatOpenAI

search = BingSearchAPIWrapper(bing_search_url="https://api.bing.microsoft.com/v7.0/search",bing_subscription_key=XXX,k=1)

prompt = ChatPromptTemplate.from_messages( [ ("system", "Have a conversation with a human. You are a helpful assistant who retrieves information from the Bing Search (the internet)."), ("user", "{input}"), MessagesPlaceholder(variable_name="agent_scratchpad"), ] )

tools = [ StructuredTool.from_function( name="BingSearch", func=search.run, description="useful for searching using Bing. " ) ]

input = "What is the apple stock price? What is the weather in berlin today?"

agent = create_openai_functions_agent(llm, tools, prompt) agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True, intermediate_steps = True) output = agent_executor.invoke({"input": input}) print(output) print('------!!!------') print(search.results(input, 1)) '''

--> Output:

{'input': 'What is the apple stock price? What is the weather in berlin today?', 'output': 'The current stock price for Apple Inc. (AAPL) is not specified in the search results, I recommend checking a reliable financial news website or stock market app for the most up-to-date information.\n\nAs for the weather in Berlin, it is currently 17° F with a RealFeel® of 16°. The air quality is poor, and there is a north-northwest wind blowing at 4 mph with gusts up to 6 mph. The sky is partly cloudy.'} ------!!!------ [{'snippet': 'Apple Inc. share price in real-time (865985 / US0378331005), charts and analyses, news, key data, turnovers, company data.', 'title': 'Apple Inc. Equity | 865985 | US0378331005 | Share Price - Börse Frankfurt', 'link': 'https://www.boerse-frankfurt.de/equity/apple-inc'}]

If the agent uses multiple sources and multiple steps, I want to know from which source each output sentence came from. Do you have some example how that can be done?

Thank you! :)

jatinchawda1503 commented 7 months ago

@glejdis, you can try providing instruction in the prompt by giving an example. 

("system", "Have a conversation with a human. You are a helpful assistant who retrieves information from the Bing Search (the internet). 
You need to provide answer with the relevant source. If you are using multiple sources and multiple steps, Mention each source answer you extracted from with urls. 
eg.
Human : What is the apple stock price? What is the weather in berlin today?
AI : The current stock price for Apple Inc. (AAPL) is not specified in the search results, I recommend checking a reliable financial news website or stock market app for the most up-to-date information.\n\nAs for the weather in Berlin, it is currently 17° F with a RealFeel® of 16°. The air quality is poor, and there is a north-northwest wind blowing at 4 mph with gusts up to 6 mph. The sky is partly cloudy.'
source 1: https://www.boerse-frankfurt.de/equity/apple-inc
source 2: https://www.apple.com/
source 3: https://www.weather.com/
...
"),
glejdis commented 7 months ago

@jatinchawda1503 that I already tried, but it does not seem to work 😅

It outputs only this:

{'input': 'What are some key advancements in the LLMs happening this week?', 'output': "Some key advancements in Large Language Models (LLMs) this week include:\n\n1. Core42’s development of an Arabic LLM named Jais 30B.\n2. NVIDIA's use of LLMs in chip design.\n3. Apple's integration of LLMs in Embodied AI.\n4. A notable collaboration between Anthropic and Google, emphasizing the importance of partnerships in driving LLM advancements.\n\nPlease note that for the most recent and detailed information, you should check the official announcements or news from these organizations.\n\nSource: Bing Search Results."}

jatinchawda1503 commented 7 months ago

@glejdis, need to do lot of trail and error :). let me check, i will get back to you, when i have something.

glejdis commented 7 months ago

@jatinchawda1503 thank you so much!

jatinchawda1503 commented 7 months ago

@glejdis, As promised, I tried different ways, Below is the closest i got. 

from langchain.agents import AgentExecutor, create_openai_functions_agent
from langchain.prompts.chat import (
    ChatPromptTemplate,
    HumanMessagePromptTemplate,
    SystemMessagePromptTemplate,
    MessagesPlaceholder
)
from langchain.tools import BaseTool, StructuredTool, tool
from langchain_community.chat_models import ChatOpenAI

search = BingSearchAPIWrapper(bing_search_url="https://api.bing.microsoft.com/v7.0/search",bing_subscription_key=bing_secret_key,k=3)
llm = ChatOpenAI(temperature=0, model="gpt-3.5-turbo-0613",openai_api_key=OPENAI_API_KEY)
prompt = ChatPromptTemplate.from_messages(
    [
        ("system", "Have a conversation with a human. You are a helpful assistant who retrieves information from the Bing Search (the internet). Provide me a Strucutred answer with valid title link and content"),
        ("user", "{input}"),
        MessagesPlaceholder(variable_name="agent_scratchpad"),
    ]
)

tools = [
    StructuredTool.from_function(
        name="BingSearch",
        func=search.results,
        description="useful for searching using Bing. "
    )
]

agent = create_openai_functions_agent(llm, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)
agent_executor.invoke({'input': 'What is the apple stock price? What is the weather in berlin today?'})
 Entering new AgentExecutor chain...

Invoking: `BingSearch` with `{'query': 'apple stock price', 'num_results': 1}`

[{'snippet': '<b>Apple</b> Inc. (AAPL) <b>Stock</b> <b>Price</b>, News, Quote &amp; History - Yahoo Finance Finance Home Watchlists My Portfolio Markets News Videos Yahoo Finance Plus Screeners Personal Finance Crypto Sectors...', 'title': 'Apple Inc. (AAPL) Stock Price, News, Quote &amp; History - Yahoo Finance', 'link': 'https://finance.yahoo.com/quote/AAPL/'}]
Invoking: `BingSearch` with `{'query': 'weather in berlin today', 'num_results': 1}`

[{'snippet': 'Current <b>Weather</b>.<b> 11:50 PM. 17° F.</b> RealFeel® 16°. Air Quality Poor. Wind NNW 4 mph. Wind Gusts 6 mph. Partly cloudy More Details.', 'title': 'Berlin, Berlin, Germany Weather Forecast | AccuWeather', 'link': 'https://www.accuweather.com/en/de/berlin/10178/weather-forecast/178087'}]The current stock price of Apple Inc. (AAPL) can be found on Yahoo Finance. You can check the stock price, news, quotes, and historical data on the following link: [Apple Inc. (AAPL) Stock Price, News, Quote & History - Yahoo Finance](https://finance.yahoo.com/quote/AAPL/).

The weather in Berlin today is partly cloudy with a temperature of 17°F (RealFeel® 16°). The air quality is poor, and the wind is blowing from the NNW at 4 mph with gusts up to 6 mph. For more details, you can visit the AccuWeather forecast for Berlin on the following link: [Berlin, Berlin, Germany Weather Forecast | AccuWeather](https://www.accuweather.com/en/de/berlin/10178/weather-forecast/178087).

> Finished chain.

Final Output

{'input': 'What is the apple stock price? What is the weather in berlin today?',
 'output': 'The current stock price of Apple Inc. (AAPL) can be found on Yahoo Finance. You can check the stock price, news, quotes, and historical data on their website. Here is the [link](https://finance.yahoo.com/quote/AAPL/) to the Apple Inc. (AAPL) stock page on Yahoo Finance.\n\nAs for the weather in Berlin today, the current temperature is 17°F with a RealFeel® of 16°F. The air quality is poor, and the wind is blowing from the NNW at 4 mph with gusts up to 6 mph. It is partly cloudy in Berlin today. You can find more details about the weather in Berlin on the AccuWeather website. Here is the [link](https://www.accuweather.com/en/de/berlin/10178/weather-forecast/178087) to the Berlin weather forecast on AccuWeather.'}
glejdis commented 7 months ago

@jatinchawda1503 that work perfectly! Thank you so much!