langchain-ai / langchain

🦜🔗 Build context-aware reasoning applications
https://python.langchain.com
MIT License
95.03k stars 15.4k forks source link

Agent using tools twice unnecessarily #2766

Closed alexiri closed 1 year ago

alexiri commented 1 year ago

I'm playing around with the Conversation Agent notebook and I realized that it tends to use a tool twice with the same input, even though it got the answer it needed the first time around.

Here's an example of that sort of behavior:

Question: Who is the current director general of CERN?

> Entering new AgentExecutor chain...
Thought: Do I need to use a tool? Yes
Action: Wikipedia
Action Input: current director general of CERN
Observation: Page: CERN
Summary: The European Organization for Nuclear Research, known as CERN (; French pronunciation: ​[sɛʁn]; Conseil européen pour la recherche nucléaire), is an intergovernmental organization that operates the largest particle physics laboratory in the world. Established in 1954, it is based in a northwestern suburb of Geneva, on the France–Switzerland border. It comprises 23 member states, and Israel (admitted in 2013) is currently the only non-European country holding full membership. CERN is an official United Nations General Assembly observer.The acronym CERN is also used to refer to the laboratory; in 2019, it had 2,660 scientific, technical, and administrative staff members, and hosted about 12,400 users from institutions in more than 70 countries. In 2016, CERN generated 49 petabytes of data.CERN's main function is to provide the particle accelerators and other infrastructure needed for high-energy physics research — consequently, numerous experiments have been constructed at CERN through international collaborations. CERN is the site of the Large Hadron Collider (LHC), the world's largest and highest-energy particle collider. The main site at Meyrin hosts a large computing facility, which is primarily used to store and analyze data from experiments, as well as simulate events. As researchers require remote access to these facilities, the lab has historically been a major wide area network hub. CERN is also the birthplace of the World Wide Web.

Page: Large Hadron Collider
Summary: The Large Hadron Collider (LHC) is the world's largest and highest-energy particle collider. It was built by the European Organization for Nuclear Research (CERN) between 1998 and 2008 in collaboration with over 10,000 scientists and hundreds of universities and laboratories, as well as more than 100 countries. It lies in a tunnel 27 kilometres (17 mi) in circumference and as deep as 175 metres (574 ft) beneath the France–Switzerland border near Geneva.
The first collisions were achieved in 2010 at an energy of 3.5 teraelectronvolts (TeV) per beam, about four times the previous world record. The discovery of the Higgs boson at the LHC was announced in 2012. Between 2013 and 2015, the LHC was shut down and upgraded; after those upgrades it reached 6.8 TeV per beam (13.6 TeV total collision energy). At the end of 2018, it was shut down for three years for further upgrades.
The collider has four crossing points where the accelerated particles collide. Seven detectors, each designed to detect different phenomena, are positioned around the crossing points. The LHC primarily collides proton beams, but it can also accelerate beams of heavy ions: lead–lead collisions and proton–lead collisions are typically performed for one month a year. 
The LHC's goal is to allow physicists to test the predictions of different theories of particle physics, including measuring the properties of the Higgs boson, searching for the large family of new particles predicted by supersymmetric theories, and other unresolved questions in particle physics.

Page: Fabiola Gianotti
Summary: Fabiola Gianotti (Italian: [faˈbiːola dʒaˈnɔtti]; born 29 October 1960) is an Italian experimental particle physicist who is the current and first woman Director-General at CERN (European Organization for Nuclear Research) in Switzerland.  Her first mandate began on 1 January 2016 and ran for a period of five years. At its 195th Session in 2019, the CERN Council selected Gianotti for a second term as Director-General. Her second five-year term began on 1 January 2021 and goes on until 2025. This is the first time in CERN's history that a Director-General has been appointed for a full second term.
Thought:Do I need to use a tool? Yes
Action: Wikipedia
Action Input: current director general of CERN
Observation: Page: CERN
Summary: The European Organization for Nuclear Research, known as CERN (; French pronunciation: ​[sɛʁn]; Conseil européen pour la recherche nucléaire), is an intergovernmental organization that operates the largest particle physics laboratory in the world. Established in 1954, it is based in a northwestern suburb of Geneva, on the France–Switzerland border. It comprises 23 member states, and Israel (admitted in 2013) is currently the only non-European country holding full membership. CERN is an official United Nations General Assembly observer.The acronym CERN is also used to refer to the laboratory; in 2019, it had 2,660 scientific, technical, and administrative staff members, and hosted about 12,400 users from institutions in more than 70 countries. In 2016, CERN generated 49 petabytes of data.CERN's main function is to provide the particle accelerators and other infrastructure needed for high-energy physics research — consequently, numerous experiments have been constructed at CERN through international collaborations. CERN is the site of the Large Hadron Collider (LHC), the world's largest and highest-energy particle collider. The main site at Meyrin hosts a large computing facility, which is primarily used to store and analyze data from experiments, as well as simulate events. As researchers require remote access to these facilities, the lab has historically been a major wide area network hub. CERN is also the birthplace of the World Wide Web.

Page: Large Hadron Collider
Summary: The Large Hadron Collider (LHC) is the world's largest and highest-energy particle collider. It was built by the European Organization for Nuclear Research (CERN) between 1998 and 2008 in collaboration with over 10,000 scientists and hundreds of universities and laboratories, as well as more than 100 countries. It lies in a tunnel 27 kilometres (17 mi) in circumference and as deep as 175 metres (574 ft) beneath the France–Switzerland border near Geneva.
The first collisions were achieved in 2010 at an energy of 3.5 teraelectronvolts (TeV) per beam, about four times the previous world record. The discovery of the Higgs boson at the LHC was announced in 2012. Between 2013 and 2015, the LHC was shut down and upgraded; after those upgrades it reached 6.8 TeV per beam (13.6 TeV total collision energy). At the end of 2018, it was shut down for three years for further upgrades.
The collider has four crossing points where the accelerated particles collide. Seven detectors, each designed to detect different phenomena, are positioned around the crossing points. The LHC primarily collides proton beams, but it can also accelerate beams of heavy ions: lead–lead collisions and proton–lead collisions are typically performed for one month a year. 
The LHC's goal is to allow physicists to test the predictions of different theories of particle physics, including measuring the properties of the Higgs boson, searching for the large family of new particles predicted by supersymmetric theories, and other unresolved questions in particle physics.

Page: Fabiola Gianotti
Summary: Fabiola Gianotti (Italian: [faˈbiːola dʒaˈnɔtti]; born 29 October 1960) is an Italian experimental particle physicist who is the current and first woman Director-General at CERN (European Organization for Nuclear Research) in Switzerland.  Her first mandate began on 1 January 2016 and ran for a period of five years. At its 195th Session in 2019, the CERN Council selected Gianotti for a second term as Director-General. Her second five-year term began on 1 January 2021 and goes on until 2025. This is the first time in CERN's history that a Director-General has been appointed for a full second term.
Thought:Do I need to use a tool? No
AI: The current Director-General of CERN is Fabiola Gianotti, an Italian experimental particle physicist who has been in the position since 2016. She was recently reappointed for a second term, which began on January 1, 2021 and will run until 2025. She is the first woman to hold the position of Director-General at CERN.

> Finished chain.
Answer: The current Director-General of CERN is Fabiola Gianotti, an Italian experimental particle physicist who has been in the position since 2016. She was recently reappointed for a second term, which began on January 1, 2021 and will run until 2025. She is the first woman to hold the position of Director-General at CERN.
ChatGPT Usage: 4673 (4540 prompt + 133 completion) in 3 requests. Total cost: $0.0093

My code is mostly equivalent to the one in the notebook, except I'm using gpt-3.5 and I added the Wikipedia tool. Any idea why it would do that?

cnhhoang850 commented 1 year ago
from langchain.agents import Tool
from langchain.agents import AgentType
from langchain.memory import ConversationBufferMemory
from langchain import OpenAI
from langchain.utilities import GoogleSearchAPIWrapper
from langchain.utilities import WikipediaAPIWrapper
from langchain.agents import initialize_agent

wikipedia = WikipediaAPIWrapper()
tools = [
    Tool(
        name = "Wikipedia",
         func=wikipedia.run,
         description="useful when you need to search anything on wikipedia"
    )
]

memory = ConversationBufferMemory(memory_key="chat_history")
llm=OpenAI(temperature=0, openai_api_key=key)
agent_chain = initialize_agent(tools, llm, agent=AgentType.CONVERSATIONAL_REACT_DESCRIPTION, verbose=True, memory=memory)

agent_chain.run('Who is the current director general of CERN?')

I use the above code to reproduce your problem but I only receive one tool use, can you post the specific code you used?

> Entering new AgentExecutor chain...

Thought: Do I need to use a tool? Yes
Action: Wikipedia
Action Input: Current director general of CERN
Observation: Page: CERN
Summary: The European Organization for Nuclear Research, known as CERN (; French pronunciation: ​[sɛʁn]; Conseil européen pour la recherche nucléaire), is an intergovernmental organization that operates the largest particle physics laboratory in the world. Established in 1954, it is based in a northwestern suburb of Geneva, on the France–Switzerland border. It comprises 23 member states, and Israel (admitted in 2013) is currently the only non-European country holding full membership. CERN is an official United Nations General Assembly observer.The acronym CERN is also used to refer to the laboratory; in 2019, it had 2,660 scientific, technical, and administrative staff members, and hosted about 12,400 users from institutions in more than 70 countries. In 2016, CERN generated 49 petabytes of data.CERN's main function is to provide the particle accelerators and other infrastructure needed for high-energy physics research — consequently, numerous experiments have been constructed at CERN through international collaborations. CERN is the site of the Large Hadron Collider (LHC), the world's largest and highest-energy particle collider. The main site at Meyrin hosts a large computing facility, which is primarily used to store and analyze data from experiments, as well as simulate events. As researchers require remote access to these facilities, the lab has historically been a major wide area network hub. CERN is also the birthplace of the World Wide Web.

Page: Fabiola Gianotti
Summary: Fabiola Gianotti (Italian: [faˈbiːola dʒaˈnɔtti]; born 29 October 1960) is an Italian experimental particle physicist who is the current and first woman Director-General at CERN (European Organization for Nuclear Research) in Switzerland.  Her first mandate began on 1 January 2016 and ran for a period of five years. At its 195th Session in 2019, the CERN Council selected Gianotti for a second term as Director-General. Her second five-year term began on 1 January 2021 and goes on until 2025. This is the first time in CERN's history that a Director-General has been appointed for a full second term.

Page: Large Hadron Collider
Summary: The Large Hadron Collider (LHC) is the world's largest and highest-energy particle collider. It was built by the European Organization for Nuclear Research (CERN) between 1998 and 2008 in collaboration with over 10,000 scientists and hundreds of universities and laboratories, as well as more than 100 countries. It lies in a tunnel 27 kilometres (17 mi) in circumference and as deep as 175 metres (574 ft) beneath the France–Switzerland border near Geneva.
The first collisions were achieved in 2010 at an energy of 3.5 teraelectronvolts (TeV) per beam, about four times the previous world record. The discovery of the Higgs boson at the LHC was announced in 2012. Between 2013 and 2015, the LHC was shut down and upgraded; after those upgrades it reached 6.8 TeV per beam (13.6 TeV total collision energy). At the end of 2018, it was shut down for three years for further upgrades.
The collider has four crossing points where the accelerated particles collide. Seven detectors, each designed to detect different phenomena, are positioned around the crossing points. The LHC primarily collides proton beams, but it can also accelerate beams of heavy ions: lead–lead collisions and proton–lead collisions are typically performed for one month a year. 
The LHC's goal is to allow physicists to test the predictions of different theories of particle physics, including measuring the properties of the Higgs boson, searching for the large family of new particles predicted by supersymmetric theories, and other unresolved questions in particle physics.

Thought: Do I need to use a tool? No
AI: The current director general of CERN is Fabiola Gianotti. She was appointed in 2016 and is the first woman to hold the position.

> Finished chain.
'The current director general of CERN is Fabiola Gianotti. She was appointed in 2016 and is the first woman to hold the position.
alexiri commented 1 year ago

hmmm... now I can't reproduce it, even though OpenAI's temperature was set to 0. That's the bad thing of working with LLMs, it's like trying to program a temperamental toddler!

dosubot[bot] commented 1 year ago

Hi, @alexiri! I'm Dosu, and I'm here to help the LangChain team manage their backlog. I wanted to let you know that we are marking this issue as stale.

From what I understand, you reported an issue with the Conversation Agent in the notebook using a tool twice with the same input, even though it already obtained the answer it needed the first time. Another user, cnhhoang850, tried to reproduce the issue but could only receive one tool use. You mentioned that you couldn't reproduce the issue anymore and also commented on the challenges of working with LLMs.

Before we close this issue, we wanted to check with you if it is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on the issue. Otherwise, feel free to close the issue yourself or it will be automatically closed in 7 days.

Thank you for your contribution to the LangChain repository!