[BUG]: Llama makes tool calling errors

[jurmy24]

Describe the bug I have provided my LLMs (including Llama 3.1 70b Instruct and 3.1 405b Instruct) with some test tools to see how well they work with the openai api sdk. While the tools sometimes work as they should, I often get responses from the chatbot like this:

<function=get_weather>{"city": "New York"}"</function>

And it does not invoke the tool_call. As it doesn't actually call any tool a new request isn't made to the api and this is what is returned to the user.

To Reproduce Just have a running chatbot with the tool calls available and ask it in natural language to eg. generate an exercise or search the textbook.

Expected behavior I expect it to call the tool with proper formatting as expected by the OpenAI SDK so that my tool call functions are invoked on the server side.

[jurmy24]

Here's an incorrect response formatting sample:

{ "content": "<function=generate_exercise>{"query": "animal grazing question for secondary school students in Tanzania"} </function>", "refusal": null, "role": "assistant", "function_call": null, "tool_calls": [] }

Here are some comments I got from Together AI regarding this issue and how to solve it:

def catch_malformed_tool(response_text: str) -> str | dict:
    """Parse response text to extract tool call if present, otherwise return original text."""
    FUNCTION_NAME = "screenpipe_search"
    TOOL_PREFIX = f"<function={FUNCTION_NAME}>"

    if not response_text.startswith(TOOL_PREFIX):
        return response_text
    # ... (logic to parse out the tool call)

Another possible solution:

• Use JSON mode to force structured output function calls and then parse it out and call the functions - this will make it so the LLM won't hallucinate or produce incorrectly formatted calls as above. - https://docs.together.ai/docs/json-mode Model: meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo