Is apply_chat_template support function call usage?

FanZhang91 commented 3 months ago

System Info

transformers= 4.42.4 python=3.10.13 ubuntu=20.04

Who can help?

No response

Information

[ ] The official example scripts
[ ] My own modified scripts

Tasks

[ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
[ ] My own task or dataset (give details below)

Reproduction

I use simple test code to verify the support of the apply_chat_template interface for the function call mode, but result shows input tools information was not added to the output prompt. Is there a problem with my usage？

from transformers import AutoTokenizer

get_current_weather = {
  "type": "function",
  "function": {
    "name": "get_current_weather",
    "description": "Get the current weather in a given location.",
    "parameters": {
        "type": "object",
        "properties": {
          "location": {
            "type": "string",
            "description": "The city and state, e.g. San Francisco, CA",
          },
          "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
        },
        "required": ["location"],
      },
  }
}

messages = [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "help me query currnet weather in San Francisco."},
    ]

tokenizer = AutoTokenizer.from_pretrained("xxx/Qwen2-1.5B-Instruct")

prompt = tokenizer.apply_chat_template(
    messages,
    tools=[get_current_weather,],
    tokenize=False,
    add_generation_prompt=True
)

print("prompt: ", prompt)

Expected behavior

None

NielsRogge commented 3 months ago

Hi,

Yes tools/function calling for apply_chat_template is supported for a few selected models. Some models which are supported (at the time of writing) include:

NousResearch/Hermes-2-Pro-Llama-3-8B
mistralai/Mixtral-8x22B-Instruct-v0.1
CohereForAI/c4ai-command-r-v01

See the docs for more info. Pinging @Rocketknight1 as there might be more.

FanZhang91 commented 3 months ago

I'm glad for your reply. The apply_chat_template function is a general function that mainly constructs an input template for LLM. In my opinion, this function should add function-related information to the prompt, such as shown below:

sys_info: {'role': 'system', 'content': 'You are a helpful assistant with access to the following functions:\n\n{"name": "get_current_weather", "description": "Get the current weather in a given location.", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "The city and state, e.g. San Francisco, CA"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["location"]}}\n\n.'}

The input parameter "tools" contains valid function-related information, but the output prompt does not contain this information. Is this normal?

JamesKCS commented 2 months ago

I am having the same problem with Mistral-7B-Instruct-v0.3. This issue causes the example on the model card not to work. The tool seems to not be included as part of the model's input. So I agree that apply_chat_template seems to not be working correctly. I am using transformers version 4.44.0.

Edit: I was using an outdated version of Mistral-7B-Instruct-v0.3. (There are different sub-versions of 0.3.)

NielsRogge commented 2 months ago

Hi,

We just officially put out a blog post on tool usage with chat templates. Read all about it here: https://huggingface.co/blog/unified-tool-use. cc @Rocketknight1

Rocketknight1 commented 2 months ago

Hi, I think there are two different issues here!

@JamesKCS, what prompt are you seeing when you run that example? It's working correctly for me. Are you sure you're using transformers 4.44.0 and loading the latest version of Mistral-7B-Instruct-v0.3, and not an older saved version of it?

Here's what I get:

>>> tool_use_prompt
'<s>[AVAILABLE_TOOLS] [{"type": "function", "function": {"name": "get_current_weather", "description": "Get the current weather", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "The city and state, e.g. San Francisco, CA"}, "format": {"type": "string", "enum": ["celsius", "fahrenheit"], "description": "The temperature unit to use. Infer this from the users location."}}, "required": ["location", "format"]}}}][/AVAILABLE_TOOLS][INST] What\'s the weather like in Paris?[/INST]'

@FanZhang91 the tool use API is currently not supported for Qwen models, but we're hoping to add that very soon, since Qwen is an important tool-use model!

JianxinMa commented 2 months ago

For Qwen models, try this unofficial template:

Step 1. Save the following template as qwen_tool_call_template.jinja:

{%- macro json_to_python_type(json_spec) %}
    {%- set basic_type_map = {
    "string": "str",
    "number": "float",
    "integer": "int",
    "boolean": "bool"
} %}
    {%- if basic_type_map[json_spec.type] is defined %}
        {{- basic_type_map[json_spec.type] }}
    {%- elif json_spec.type == "array" %}
        {{- "list[" +  json_to_python_type(json_spec|items) + "]" }}
    {%- elif json_spec.type == "object" %}
        {%- if json_spec.additionalProperties is defined %}
            {{- "dict[str, " + json_to_python_type(json_spec.additionalProperties) + ']' }}
        {%- else %}
            {{- "dict" }}
        {%- endif %}
    {%- elif json_spec.type is iterable %}
        {{- "Union[" }}
        {%- for t in json_spec.type %}
            {{- json_to_python_type({"type": t}) }}
            {%- if not loop.last %}
                {{- "," }}
            {%- endif %}
        {%- endfor %}
        {{- "]" }}
    {%- else %}
        {{- "Any" }}
    {%- endif %}
{%- endmacro %}

{%- if tools %}
    {{- '<|im_start|>system\n' }}
    {%- if messages[0]['role'] == 'system' %}
        {{- messages[0]['content'] + '\n\n' }}
    {%- endif %}
    {{- '# Tools\n\n' }}
    {{- "You are a function calling AI model. You are provided with function signatures within <tools></tools> XML tags. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. Here are the available tools: <tools> " }}
    {%- for tool in tools %}
        {%- if tool.function is defined %}
            {%- set tool = tool.function %}
        {%- endif %}
        {{- '{"type": "function", "function": ' }}
        {{- '{"name": ' + tool.name + '", ' }}
        {{- '"description": "' + tool.name + '(' }}
        {%- for param_name, param_fields in tool.parameters.properties|items %}
            {{- param_name + ": " + json_to_python_type(param_fields) }}
            {%- if not loop.last %}
                {{- ", " }}
            {%- endif %}
        {%- endfor %}
        {{- ")" }}
        {%- if tool.return is defined %}
            {{- " -> " + json_to_python_type(tool.return) }}
        {%- endif %}
        {{- " - " + tool.description + "\n\n" }}
        {%- for param_name, param_fields in tool.parameters.properties|items %}
            {%- if loop.first %}
                {{- "    Args:\n" }}
            {%- endif %}
            {{- "        " + param_name + "(" + json_to_python_type(param_fields) + "): " + param_fields.description|trim }}
        {%- endfor %}
        {%- if tool.return is defined and tool.return.description is defined %}
            {{- "\n    Returns:\n        " + tool.return.description }}
        {%- endif %}
        {{- '"' }}
        {{- ', "parameters": ' }}
        {%- if tool.parameters.properties | length == 0 %}
            {{- "{}" }}
        {%- else %}
            {{- tool.parameters|tojson }}
        {%- endif %}
        {{- "}" }}
        {%- if not loop.last %}
            {{- "\n" }}
        {%- endif %}
    {%- endfor %}
    {{- " </tools>" }}
    {{- 'Use the following pydantic model json schema for each tool call you will make: {"properties": {"arguments": {"title": "Arguments", "type": "object"}, "name": {"title": "Name", "type": "string"}}, "required": ["arguments", "name"], "title": "FunctionCall", "type": "object"}
' }}
    {{- "For each function call return a json object with function name and arguments within <tool_call></tool_call> XML tags as follows:
" }}
    {{- "<tool_call>
" }}
    {{- '{"name": <function-name>, "arguments": <args-json-object>}
' }}
    {{- '</tool_call><|im_end|>\n' }}
{%- else %}
    {%- if messages[0]['role'] != 'system' %}
        {{- '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n' }}
    {%- else %}
        {{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
    {%- endif %}
{%- endif %}
{%- for message in messages %}
    {%- if message.role == "user" or (message.role == "system" and not loop.first) or (message.role == "assistant" and message.tool_calls is not defined) %}
        {{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
    {%- elif message.role == "assistant" %}
        {{- '<|im_start|>' + message.role + '\n<tool_call>\n' }}
        {%- for tool_call in message.tool_calls %}
            {%- if tool_call.function is defined %}
                {%- set tool_call = tool_call.function %}
            {%- endif %}
            {{- '{' }}
            {{- '"name": "' }}
            {{- tool_call.name }}
            {%- if tool_call.arguments is defined %}
                {{- ', ' }}
                {{- '"arguments": ' }}
                {{- tool_call.arguments|tojson }}
            {%- endif %}
            {{- '"}' }}
            {{- '\n</tool_call>' }}
        {%- endfor %}
        {{- '<|im_end|>\n' }}
    {%- elif message.role == "tool" %}
        {%- if not message.name is defined %}
            {{- raise_exception("Tool response dicts require a 'name' key indicating the name of the called function!") }}
        {%- endif %}
        {{- '<|im_start|>user\n<tool_response>\n' }}
        {{- '{"name": "' }}
        {{- message.name }}
        {{- '", "content": ' }}
        {{- message.content|tojson + '}' }}
        {{- '\n</tool_response><|im_end|>\n' }}
    {%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
    {{- '<|im_start|>assistant\n' }}
{%- endif %}

Step 2. Then you can test it like this:

from transformers import AutoModelForCausalLM, AutoTokenizer

def get_current_temperature(location: str, unit: str) -> float:
    """
    Get the current temperature at a location.

    Args:
        location: The location to get the temperature for, in the format "City, Country"
        unit: The unit to return the temperature in. (choices: ["celsius", "fahrenheit"])
    Returns:
        The current temperature at the specified location in the specified units, as a float.
    """
    return 22.  # A real function should probably actually get the temperature!

def get_current_wind_speed(location: str) -> float:
    """
    Get the current wind speed in km/h at a given location.

    Args:
        location: The location to get the temperature for, in the format "City, Country"
    Returns:
        The current wind speed at the given location in km/h, as a float.
    """
    return 6.  # A real function should probably actually get the wind speed!

model_name = 'Qwen/Qwen2-0.5B-Instruct'
model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

with open('qwen_tool_call_template.jinja') as fin:
    tokenizer.chat_template = fin.read()

def chat(messages):
    print('#DEBUG BEGIN')
    print(
        tokenizer.apply_chat_template(
            messages,
            tools=[get_current_temperature, get_current_wind_speed],
            add_generation_prompt=True,
            tokenize=False,
        ))
    print('#DEBUG END')

    inputs = tokenizer.apply_chat_template(
        messages,
        tools=[get_current_temperature, get_current_wind_speed],
        add_generation_prompt=True,
        return_dict=True,
        return_tensors='pt',
    )
    inputs = {k: v.to(model.device) for k, v in inputs.items()}
    out = model.generate(**inputs, max_new_tokens=1024)
    print('#OUTPUT BEGIN')
    print(tokenizer.decode(out[0][len(inputs['input_ids'][0]):]))
    print('#OUTPUT END')

def main():
    messages = [
        {
            'role':
                'system',
            'content':
                'You are a bot that responds to weather queries. You should reply with the unit used in the queried location.'
        },
        {
            'role': 'user',
            'content': "Hey, what's the temperature in Paris right now?"
        },
    ]
    chat(messages)

    # Please read
    #   https://huggingface.co/docs/transformers/main/en/chat_templating#advanced-tool-use--function-calling
    # to understand what I'm doing here.

    tool_call_id = 'vAHdf3'  # Random ID, should be unique for each tool call
    tool_call = {'name': 'get_current_temperature', 'arguments': {'location': 'Paris, France', 'unit': 'celsius'}}
    messages.append({
        'role': 'assistant',
        'content': '',
        'tool_calls': [{
            'id': tool_call_id,
            'type': 'function',
            'function': tool_call
        }]
    })
    messages.append({
        'role': 'tool',
        'tool_call_id': tool_call_id,
        'name': 'get_current_temperature',
        'content': '22.0'
    })
    chat(messages)

if __name__ == '__main__':
    main()

JamesKCS commented 2 months ago

@Rocketknight1 Thank you for your reply. I did not did not realize that there were multiple versions of Mistral-7B-Instruct-v0.3, and I was using an outdated version. Using the most up-to-date version solved it for me. Thank you again.

Rocketknight1 commented 2 months ago

@JianxinMa amazing work on the template! Can you open a PR to add it as the official chat template to the Qwen model repos? You can just do something like this:


tokenizer = AutoTokenizer.from_pretrained(model_name)

with open('qwen_tool_call_template.jinja') as fin:
    tokenizer.chat_template = fin.read()

tokenizer.push_to_hub(model_name, create_pr=True)

If you open those PRs, please ping me (my username is Rocketknight1 on the HF Hub as well), and I'll help with reviewing them. I spoke to a couple of other people at Hugging Face about this, and we're excited to get proper tool-calling support for Qwen!

MrRace commented 2 months ago

looking forward to the Qwen model being supported as soon as possible~

github-actions[bot] commented 1 month ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

huggingface / transformers