Open FanZhang91 opened 1 month ago
NielsRogge
commented
1 month ago Hi,
Yes tools/function calling for apply_chat_template is supported for a few selected models. Some models which are supported (at the time of writing) include:
See the docs for more info. Pinging @Rocketknight1 as there might be more.
FanZhang91
commented
1 month ago I'm glad for your reply. The apply_chat_template function is a general function that mainly constructs an input template for LLM. In my opinion, this function should add function-related information to the prompt, such as shown below:
sys_info: {'role': 'system', 'content': 'You are a helpful assistant with access to the following functions:\n\n{"name": "get_current_weather", "description": "Get the current weather in a given location.", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "The city and state, e.g. San Francisco, CA"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["location"]}}\n\n.'}The input parameter "tools" contains valid function-related information, but the output prompt does not contain this information. Is this normal?
JamesKCS
commented
3 weeks ago I am having the same problem with Mistral-7B-Instruct-v0.3. This issue causes the example on the model card not to work. The tool seems to not be included as part of the model's input. So I agree that apply_chat_template seems to not be working correctly. I am using transformers version 4.44.0.
Edit: I was using an outdated version of Mistral-7B-Instruct-v0.3. (There are different sub-versions of 0.3.)
NielsRogge
commented
3 weeks ago Hi,
We just officially put out a blog post on tool usage with chat templates. Read all about it here: https://huggingface.co/blog/unified-tool-use. cc @Rocketknight1
Rocketknight1
commented
3 weeks ago Hi, I think there are two different issues here!
@JamesKCS, what prompt are you seeing when you run that example? It's working correctly for me. Are you sure you're using transformers 4.44.0 and loading the latest version of Mistral-7B-Instruct-v0.3, and not an older saved version of it?
Here's what I get:
>>> tool_use_prompt
'<s>[AVAILABLE_TOOLS] [{"type": "function", "function": {"name": "get_current_weather", "description": "Get the current weather", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "The city and state, e.g. San Francisco, CA"}, "format": {"type": "string", "enum": ["celsius", "fahrenheit"], "description": "The temperature unit to use. Infer this from the users location."}}, "required": ["location", "format"]}}}][/AVAILABLE_TOOLS][INST] What\'s the weather like in Paris?[/INST]'@FanZhang91 the tool use API is currently not supported for Qwen models, but we're hoping to add that very soon, since Qwen is an important tool-use model!
JianxinMa
commented
3 weeks ago For Qwen models, try this unofficial template:
Step 1. Save the following template as qwen_tool_call_template.jinja:
{%- macro json_to_python_type(json_spec) %}
{%- set basic_type_map = {
"string": "str",
"number": "float",
"integer": "int",
"boolean": "bool"
} %}
{%- if basic_type_map[json_spec.type] is defined %}
{{- basic_type_map[json_spec.type] }}
{%- elif json_spec.type == "array" %}
{{- "list[" + json_to_python_type(json_spec|items) + "]" }}
{%- elif json_spec.type == "object" %}
{%- if json_spec.additionalProperties is defined %}
{{- "dict[str, " + json_to_python_type(json_spec.additionalProperties) + ']' }}
{%- else %}
{{- "dict" }}
{%- endif %}
{%- elif json_spec.type is iterable %}
{{- "Union[" }}
{%- for t in json_spec.type %}
{{- json_to_python_type({"type": t}) }}
{%- if not loop.last %}
{{- "," }}
{%- endif %}
{%- endfor %}
{{- "]" }}
{%- else %}
{{- "Any" }}
{%- endif %}
{%- endmacro %}
{%- if tools %}
{{- '<|im_start|>system\n' }}
{%- if messages[0]['role'] == 'system' %}
{{- messages[0]['content'] + '\n\n' }}
{%- endif %}
{{- '# Tools\n\n' }}
{{- "You are a function calling AI model. You are provided with function signatures within <tools></tools> XML tags. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. Here are the available tools: <tools> " }}
{%- for tool in tools %}
{%- if tool.function is defined %}
{%- set tool = tool.function %}
{%- endif %}
{{- '{"type": "function", "function": ' }}
{{- '{"name": ' + tool.name + '", ' }}
{{- '"description": "' + tool.name + '(' }}
{%- for param_name, param_fields in tool.parameters.properties|items %}
{{- param_name + ": " + json_to_python_type(param_fields) }}
{%- if not loop.last %}
{{- ", " }}
{%- endif %}
{%- endfor %}
{{- ")" }}
{%- if tool.return is defined %}
{{- " -> " + json_to_python_type(tool.return) }}
{%- endif %}
{{- " - " + tool.description + "\n\n" }}
{%- for param_name, param_fields in tool.parameters.properties|items %}
{%- if loop.first %}
{{- " Args:\n" }}
{%- endif %}
{{- " " + param_name + "(" + json_to_python_type(param_fields) + "): " + param_fields.description|trim }}
{%- endfor %}
{%- if tool.return is defined and tool.return.description is defined %}
{{- "\n Returns:\n " + tool.return.description }}
{%- endif %}
{{- '"' }}
{{- ', "parameters": ' }}
{%- if tool.parameters.properties | length == 0 %}
{{- "{}" }}
{%- else %}
{{- tool.parameters|tojson }}
{%- endif %}
{{- "}" }}
{%- if not loop.last %}
{{- "\n" }}
{%- endif %}
{%- endfor %}
{{- " </tools>" }}
{{- 'Use the following pydantic model json schema for each tool call you will make: {"properties": {"arguments": {"title": "Arguments", "type": "object"}, "name": {"title": "Name", "type": "string"}}, "required": ["arguments", "name"], "title": "FunctionCall", "type": "object"}
' }}
{{- "For each function call return a json object with function name and arguments within <tool_call></tool_call> XML tags as follows:
" }}
{{- "<tool_call>
" }}
{{- '{"name": <function-name>, "arguments": <args-json-object>}
' }}
{{- '</tool_call><|im_end|>\n' }}
{%- else %}
{%- if messages[0]['role'] != 'system' %}
{{- '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n' }}
{%- else %}
{{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- for message in messages %}
{%- if message.role == "user" or (message.role == "system" and not loop.first) or (message.role == "assistant" and message.tool_calls is not defined) %}
{{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
{%- elif message.role == "assistant" %}
{{- '<|im_start|>' + message.role + '\n<tool_call>\n' }}
{%- for tool_call in message.tool_calls %}
{%- if tool_call.function is defined %}
{%- set tool_call = tool_call.function %}
{%- endif %}
{{- '{' }}
{{- '"name": "' }}
{{- tool_call.name }}
{%- if tool_call.arguments is defined %}
{{- ', ' }}
{{- '"arguments": ' }}
{{- tool_call.arguments|tojson }}
{%- endif %}
{{- '"}' }}
{{- '\n</tool_call>' }}
{%- endfor %}
{{- '<|im_end|>\n' }}
{%- elif message.role == "tool" %}
{%- if not message.name is defined %}
{{- raise_exception("Tool response dicts require a 'name' key indicating the name of the called function!") }}
{%- endif %}
{{- '<|im_start|>user\n<tool_response>\n' }}
{{- '{"name": "' }}
{{- message.name }}
{{- '", "content": ' }}
{{- message.content|tojson + '}' }}
{{- '\n</tool_response><|im_end|>\n' }}
{%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
{{- '<|im_start|>assistant\n' }}
{%- endif %}Step 2. Then you can test it like this:
from transformers import AutoModelForCausalLM, AutoTokenizer
def get_current_temperature(location: str, unit: str) -> float:
"""
Get the current temperature at a location.
Args:
location: The location to get the temperature for, in the format "City, Country"
unit: The unit to return the temperature in. (choices: ["celsius", "fahrenheit"])
Returns:
The current temperature at the specified location in the specified units, as a float.
"""
return 22. # A real function should probably actually get the temperature!
def get_current_wind_speed(location: str) -> float:
"""
Get the current wind speed in km/h at a given location.
Args:
location: The location to get the temperature for, in the format "City, Country"
Returns:
The current wind speed at the given location in km/h, as a float.
"""
return 6. # A real function should probably actually get the wind speed!
model_name = 'Qwen/Qwen2-0.5B-Instruct'
model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
with open('qwen_tool_call_template.jinja') as fin:
tokenizer.chat_template = fin.read()
def chat(messages):
print('#DEBUG BEGIN')
print(
tokenizer.apply_chat_template(
messages,
tools=[get_current_temperature, get_current_wind_speed],
add_generation_prompt=True,
tokenize=False,
))
print('#DEBUG END')
inputs = tokenizer.apply_chat_template(
messages,
tools=[get_current_temperature, get_current_wind_speed],
add_generation_prompt=True,
return_dict=True,
return_tensors='pt',
)
inputs = {k: v.to(model.device) for k, v in inputs.items()}
out = model.generate(**inputs, max_new_tokens=1024)
print('#OUTPUT BEGIN')
print(tokenizer.decode(out[0][len(inputs['input_ids'][0]):]))
print('#OUTPUT END')
def main():
messages = [
{
'role':
'system',
'content':
'You are a bot that responds to weather queries. You should reply with the unit used in the queried location.'
},
{
'role': 'user',
'content': "Hey, what's the temperature in Paris right now?"
},
]
chat(messages)
# Please read
# https://huggingface.co/docs/transformers/main/en/chat_templating#advanced-tool-use--function-calling
# to understand what I'm doing here.
tool_call_id = 'vAHdf3' # Random ID, should be unique for each tool call
tool_call = {'name': 'get_current_temperature', 'arguments': {'location': 'Paris, France', 'unit': 'celsius'}}
messages.append({
'role': 'assistant',
'content': '',
'tool_calls': [{
'id': tool_call_id,
'type': 'function',
'function': tool_call
}]
})
messages.append({
'role': 'tool',
'tool_call_id': tool_call_id,
'name': 'get_current_temperature',
'content': '22.0'
})
chat(messages)
if __name__ == '__main__':
main()Related: https://github.com/QwenLM/Qwen2/issues/805#issuecomment-2258310024
JamesKCS
commented
3 weeks ago @Rocketknight1 Thank you for your reply. I did not did not realize that there were multiple versions of Mistral-7B-Instruct-v0.3, and I was using an outdated version. Using the most up-to-date version solved it for me. Thank you again.
Rocketknight1
commented
3 weeks ago @JianxinMa amazing work on the template! Can you open a PR to add it as the official chat template to the Qwen model repos? You can just do something like this:
tokenizer = AutoTokenizer.from_pretrained(model_name)
with open('qwen_tool_call_template.jinja') as fin:
tokenizer.chat_template = fin.read()
tokenizer.push_to_hub(model_name, create_pr=True)If you open those PRs, please ping me (my username is Rocketknight1 on the HF Hub as well), and I'll help with reviewing them. I spoke to a couple of other people at Hugging Face about this, and we're excited to get proper tool-calling support for Qwen!
MrRace
commented
1 week ago looking forward to the Qwen model being supported as soon as possible~
System Info
transformers= 4.42.4 python=3.10.13 ubuntu=20.04
Who can help?
No response
Information
Tasks
examplesfolder (such as GLUE/SQuAD, ...)Reproduction
I use simple test code to verify the support of the apply_chat_template interface for the function call mode, but result shows input tools information was not added to the output prompt. Is there a problem with my usage？
prompt: <|im_start|>system You are a helpful assistant.<|im_end|> <|im_start|>user help me query currnet weather in San Francisco.<|im_end|> <|im_start|>assistant
Expected behavior
None