NousResearch / Hermes-Function-Calling

MIT License
475 stars 74 forks source link

Model doesn't detect when there is no need for any tools. #15

Closed unclecode closed 1 week ago

unclecode commented 3 months ago

In this case there is no need to call any function, but it returns duckduckgo_news!

schema = {'type': 'function',
 'function': {'name': 'duckduckgo_news',
  'description': '"""\nGet a search phrase and search on news using DuckDuckGo, returning a list of JSON results.\n\nParameters:\n    query (str): The search query.\n    max_results (int, optional): The maximum number of results to return. Defaults to None, which returns all available results.\n\nReturns:\n    dict: A dictionary with the key \'response\' containing a list of dictionaries, where each dictionary represents a news result.\n"""\nfrom duckduckgo_search import DDGS\n\ndef duckduckgo_news(query, max_results=None):\n    if not query:\n        raise ValueError("Query cannot be empty")\n    with DDGS() as ddgs:\n        results = [r for r in ddgs.news(query, safesearch=\'off\', max_results=max_results)]\n    return {\'response\': results}',
  'parameters': {'type': 'object',
   'properties': {'query': {'type': 'string',
     'description': 'The search query to use for the news search.'},
    'max_results': {'type': 'integer',
     'description': 'The maximum number of results to return. If not provided, all available results will be returned.',
     'default': None}},
   'required': ['query']}}}

response = ollama.chat(
    model = "adrienbrault/nous-hermes2pro:Q8_0-tools",
    messages=[
        {
            "role": "user",
            "content": f"""<tools>{json.dumps(schema)}</tools>\n\nHow r you?""",
        },
    ],
)

print(response["message"]["content"])

It returns:

<tool_call>\
{"arguments": {"query": "how are you", "max_results": 1}, "name": "duckduckgo_news"}\
</tool_call>
interstellarninja commented 3 months ago

hmm that's interesting, it hasn't happened ever in our testing unless the query is semantically close to the function descriptions.

adrienbrault/nous-hermes2pro:Q8_0-tools model is using the following system prompt. you may edit the system prompt to not call any function when it's not necessary.

You are a function calling AI model. You are provided with function signatures within <tools></tools> XML tags.
You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions.
Use the following json schema for each tool call you will make: {"title": "FunctionCall", "type": "object", "properties": {"arguments": {"title": "Arguments", "type": "object"}, "name": {"title": "Name", "type": "string"}}, "required": ["arguments", "name"]}
For each function call return a json object with function name and arguments within <tool_call></tool_call> XML tags as follows:
<tool_call>
{"arguments": <args-dict>, "name": <function-name>}
</tool_call>
teknium1 commented 1 week ago

Will be fixed in hermes 3