[BUG] Problem with new Wikipedia Search functionality

HalberdOfPineapple commented 1 year ago

Required prerequisites

[X] I have read the documentation https://camel-ai.github.io/camel/camel.html.
[X] I have searched the Issue Tracker and Discussions that this hasn't already been reported. (+1 or comment there if it has.)
[X] Consider asking first in a Discussion.

What version of camel are you using?

0.1.0

System information

3.10.10 (main, Mar 21 2023, 18:45:11) [GCC 11.2.0] linux 0.1.0

Problem description

There are some problems observed in running role_playing_with_function_call.py with using the module camel/functions/search_functions.py updated in #253.

To ensure the reproducibility, I set the temperature to be 0.0 in the example (though there can still occur some variations between different executions).

Firstly, we now have two functions performing almost the same functionality of searching information in Wikipedia but I have only observed the agent uses the original one (search_wiki, implemented by hand).

Thus, to test the effect of the new function search_wiki_by_api, I removed the original search_wiki from SEARCH_FUNCS making the model only aware of the new function. Here comes a problem that the user becomes "idle", it can only output the single response "Apologies for the confusion, but as an AI, I'm ready to receive your instructions. Please proceed with the task., as shown below:


AI User:

Apologies for the confusion, but as an AI, I'm ready to receive your instructions. Please proceed with the task.

AI Assistant:

Solution: Based on the information obtained from Wikipedia, KAIST (Korea Advanced Institute of Science and Technology) was established in 1971. Given that the current year is 2023, we can calculate KAIST's age by subtracting 1971 from 2023, which gives us 52 years. Adding 10 more years to this age, we get 62 years. Therefore, KAIST will be 62 years old in 10 years from 2023. Next request.

AI User:

Apologies for the confusion, but as an AI, I'm ready to receive your instructions. Please proceed with the task.

AI Assistant:

Function Execution: search_wiki_by_api
        Args: {'entity': 'King Abdullah University of Science and Technology'}
        Result: King Abdullah University of Science and Technology (KAUST; Arabic: جامعة الملك عبد الله للعلوم و التقنية jāmiʿat al-malik ʿabd al-Lāh li-l-ʿulūm wa-t-teqniyya) is a public research university located in Thuwal, Saudi Arabia. Founded in 2009, the university provides research and graduate
training programs in English as the official language of instruction.
KAUST is the first mixed-gender university campus in Saudi Arabia. In 2013, the university was among the 500 fastest growing research and citation records in the world. In the 2016 Nature Index Rising Stars, the university ranked 19th in the world of the fastest rising universities for high quality research output.

Function Execution: sub
        Args: {'a': 2023, 'b': 2009}
        Result: 14

Function Execution: add
        Args: {'a': 14, 'b': 10}
        Result: 24

Solution: Based on the information obtained from Wikipedia, King Abdullah University of Science and Technology (KAUST) was established in 2009. Given that the current year is 2023, we can calculate KAUST's age by subtracting 2009 from 2023, which gives us 14 years. Adding 10 more years to this age, we get 24 years. Therefore, KAUST will be 24 years old in 10 years from 2023. Next request.

AI User:

Apologies for the confusion, but as an AI, I'm ready to receive your instructions. Please proceed with the task.

AI Assistant:

Solution: Based on the information obtained from Wikipedia, King Abdullah University of Science and Technology (KAUST) was established in 2009. Given that the current year is 2023, we can calculate KAUST's age by subtracting 2009 from 2023, which gives us 14 years. Adding 10 more years to this age, we get 24 years. Therefore, KAUST will be 24 years old in 10 years from 2023. Next request.

AI User:

Apologies for the confusion, but as an AI, I'm ready to receive your instructions. Please proceed with the task.

AI Assistant:

Solution: Based on the information obtained from Wikipedia, King Abdullah University of Science and Technology (KAUST) was established in 2009. Given that the current year is 2023, we can calculate KAUST's age by subtracting 2009 from 2023, which gives us 14 years. Adding 10 more years to this age, we get 24 years. Therefore, KAUST will be 24 years old in 10 years from 2023. Next request.

AI User:

Apologies for the confusion, but as an AI, I'm ready to receive your instructions. Please proceed with the task.

AI Assistant:

Solution: Based on the information obtained from Wikipedia, King Abdullah University of Science and Technology (KAUST) was established in 2009. Given that the current year is 2023, we can calculate KAUST's age by subtracting 2009 from 2023, which gives us 14 years. Adding 10 more years to this age, we get 24 years. Therefore, KAUST will be 24 years old in 10 years from 2023. Next request.

AI User:

Apologies for the confusion, but as an AI, I'm ready to receive your instructions. Please proceed with the task.

AI Assistant:

Solution: Based on the information obtained from Wikipedia, King Abdullah University of Science and Technology (KAUST) was established in 2009. Given that the current year is 2023, we can calculate KAUST's age by subtracting 2009 from 2023, which gives us 14 years. Adding 10 more years to this age, we get 24 years. Therefore, KAUST will be 24 years old in 10 years from 2023. Next request.

where the execution was stuck into an infinite loop. And the first reponse returned by the assistant was incorrect, which did not call the search function and was about KAIST instead of KAUST.

I suspect the problem arises from that the annotations (note that the annotations are essentially the prompt to explain the function to the model) given to the new function search_wiki_by_api may be too implementation-related. And I replaced the annotation of it to be:

def search_wiki_by_api(entity: str) -> str:
    r"""Search the entity in WikiPedia and return the summary of the
    required page, containing factual information about the given entity.

    Args:
        entity (string): The entity to be searched.

    Returns:
        string: The search result. If the page corresponding to the entity
            exists, return the summary of this entity in a string.
    """

After this I then called the role_playing_with_function_call.py, this time the function calling can be correctly called without the user being idle. But the problem is that, when "KAUST" is the entity word to be searched, the returned result by wikipedia API (wikipedia.summary) is the summary about KAIST like below:

AI User:

Instruction: Find out the year KAUST was founded.
Input: None

AI Assistant:

Function Execution: search_wiki_by_api
        Args: {'entity': 'KAUST'}
        Result: The Korea Advanced Institute of Science and Technology (KAIST) is a national research university located in Daedeok Innopolis, Daejeon,
South Korea. KAIST was established by the Korean government in 1971 as the nation's first public, research-oriented science and engineering institution. KAIST is considered to be one of the most prestigious universities in the nation. KAIST has been internationally accredited in business education, and
hosting the Secretariat of the Association of Asia-Pacific Business Schools (AAPBS). KAIST has 10,504 full-time students and 1,342 faculty researchers (as of Fall 2019 Semester) and had a total budget of US$765 million in 2013, of which US$459 million was from research contracts.

Solution: KAUST, the Korea Advanced Institute of Science and Technology, was founded in the year 1971. This information was obtained by searching for the entity 'KAUST' using the 'search_wiki_by_api' function. The result returned a summary of the Wikipedia page for KAIST, which states that it was established by the Korean government in 1971.

Next request.

AI User:

I apologize for the confusion, but KAUST refers to King Abdullah University of Science and Technology in Saudi Arabia, not the Korea Advanced Institute
of Science and Technology. The King Abdullah University of Science and Technology was founded in 2009.

Next instruction, please.

Reproducible example code

The Python snippets:

    function_list = [*MATH_FUNCS, *SEARCH_FUNCS]
    assistant_model_config = FunctionCallingConfig.from_openai_function_list(
        function_list=function_list,
        kwargs=dict(temperature=0),
    )

    role_play_session = RolePlaying(
        assistant_role_name="Searcher",
        user_role_name="Professor",
        assistant_agent_kwargs=dict(
            model=model_type,
            model_config=assistant_model_config,
            function_list=function_list,
        ),
        user_agent_kwargs=dict(model=model_type),
        task_prompt=task_prompt,
        with_task_specify=False,
        task_specify_agent_kwargs=dict(model=model_type),
    )

Command lines:

python examples/function_call/role_playing_with_function.py

Steps to reproduce:

pip install -e .
python examples/function_call/role_playing_with_function.py

Traceback

No response

Expected behavior

No response

Additional context

No response

HalberdOfPineapple commented 1 year ago

@lightaime @zhiyu-01 Could you have a check on this?

zhiyu-01 commented 1 year ago

@lightaime @zhiyu-01 Could you have a check on this?

It may be caused by its default arguments: redirect=True, which redirect "KAUST" to "KAIST" and "KAIST" to "GIST". When I set redirect=False, it deals "KAIST" correctly, but search "KAUST" returns an error: An error occurred: "KAUST" resulted in a redirect. Set the redirect property to True to allow automatic redirects. I will remove the function "search_wiki_by_api" and try to fix it.

HalberdOfPineapple commented 1 year ago

@lightaime @zhiyu-01 Could you have a check on this?

It may be caused by its default arguments: redirect=True, which redirect "KAUST" to "KAIST" and "KAIST" to "GIST". When I set redirect=False, it deals "KAIST" correctly, but search "KAUST" returns an error: An error occurred: "KAUST" resulted in a redirect. Set the redirect property to True to allow automatic redirects. I will remove the function "search_wiki_by_api" and try to fix it.

Thanks for your reply. I just found the reason should lie in auto_suggest as said in https://stackoverflow.com/questions/66630711/the-wikipedia-api-seems-to-almost-always-get-the-word-in-question-wrong. I have modified a version and will push to a new branch soon.

zhiyu-01 commented 1 year ago

@lightaime @zhiyu-01 Could you have a check on this?

It may be caused by its default arguments: redirect=True, which redirect "KAUST" to "KAIST" and "KAIST" to "GIST". When I set redirect=False, it deals "KAIST" correctly, but search "KAUST" returns an error: An error occurred: "KAUST" resulted in a redirect. Set the redirect property to True to allow automatic redirects. I will remove the function "search_wiki_by_api" and try to fix it.

Thanks for your reply. The reason should lie in auto_suggest as said in https://stackoverflow.com/questions/66630711/the-wikipedia-api-seems-to-almost-always-get-the-word-in-question-wrong. I have modified a version and will push to a new branch soon. This is my new code: result = wikipedia.search(entity) print(result) result = wikipedia.summary(result[0], sentences=5, redirect=False, auto_suggest=False) The result is: ['King Abdullah University of Science and Technology', 'Shaheen (supercomputer)', 'Thuwal', 'Saudi Arabia', 'Nadhmi Al-Nasr', 'Marc G. Genton', 'Jürgen Schmidhuber', 'Lubna Olayan', 'Amin H. Nasser', 'Koch snowflake']

King Abdullah University of Science and Technology (KAUST; Arabic: جامعة الملك عبد الله للعلوم و التقنية jāmiʿat al-malik ʿabd al-Lāh li-l-ʿulūm wa-t-teqniyya) is a public research university located in Thuwal, Saudi Arabia. Founded in 2009, the university provides research and graduate training programs in English as the official language of instruction. KAUST is the first mixed-gender university campus in Saudi Arabia. In 2013, the university was among the 500 fastest growing research and citation records in the world. In the 2016 Nature Index Rising Stars, the university ranked 19th in the world of the fastest rising universities for high quality research output.

Maybe you want take consider of the wikipedia.search() function, it will give a list contains what wikipedia can get.

HalberdOfPineapple commented 1 year ago

Maybe you want take consider of the wikipedia.search() function, it will give a list contains what wikipedia can get.

True I have considered this. But it is a bit of tedious to iterate through the list. Actually the redirect can be kept True to solve the ambiguity redirection. The original problem lies in auto_suggest can directly change the search word (such as KAUST -> KAIST)

zhiyu-01 commented 1 year ago

I just tried it, it looks good.

camel-ai / camel