Closed ArjunBhalla98 closed 1 week ago
Thanks @drbh for taking this 🙌
Thanks for picking this up all! Just wanted to add some more info I found from tinkering around. Interestingly, I seem to get different responses based on the tool choice.
If I use tool_choice=<name-of-tool>
, this works fine (or at least, will give a "descriptive" EoF error as mentioned in the original issue). However, if I use tool_choice="auto"
, I will always get EOF\n
. Specifically:
Prompt: "\<s>[INST] To better understand the volatility and risk associated with this particular stock, I need to calculate the standard deviation of its daily closing prices over the past 10 trading days. Here are the figures I've gathered: 1000, 2000, 3000, 4000, 5000, 7000, 9000, 15000, 20000, and 30000. Can you provide me with the standard deviation for these closing prices? [/INST]"
With tool_choice=<name-of-tool>
:
Response: Response: {'object': 'chat.completion', 'id': '', 'created': 1721167846, 'model': '/mnt/models', 'system_fingerprint': '2.1.1-sha-4dfdb48', 'choices': [{'index': 0, 'message': {'role': 'assistant', 'tool_calls': [{'id': '0', 'type': 'function', 'function': {'description': None, 'name': 'stdev', 'arguments': {'numbers': [1000, 2000, 3000, 4000, 5000, 7000, 9000, 15000, 20000, 30000]}}}]}, 'logprobs': None, 'finish_reason': 'eos_token'}], 'usage': {'prompt_tokens': 141, 'completion_tokens': 94, 'total_tokens': 235}}
With tool_choice="auto"
:
Response: b'EOF\n'
I tried this across the sub sample of the dataset mentioned above and this appears to be consistent across all samples, i.e. that every response, regardless of what it was prior when the issue was opened, became EOF\n
.
Thanks again.
Hi @ArjunBhalla98 thank you for noting this issue. I've just opened https://github.com/huggingface/text-generation-inference/pull/2244 with some improvements to grammar/tools and specifically a fix to a bug with the tool_choice="auto"
value. Additionally the PR adds better error messages in the case the that it fails to parse the generated text to JSON. I believe these changes should resolve your EOF issue once merged.
Thank you very much! I will keep an eye on the PR and when it is merged I will update and try it out on our end.
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days.
I think this issue was fixed by #2244? 🤔
Hi @ArjunBhalla98 thank you for opening this issue and https://github.com/huggingface/text-generation-inference/issues/2310, these issues should all be resolved now with the recent improvements to grammars/tools. Going to close this similar to https://github.com/huggingface/text-generation-inference/issues/2310 but please feel free to reopen if these issues can be reproduced on main. Thank you!!
System Info
Privately hosted instance of TGI Version: 2.1.1
Deployed as a standalone kserve predictor Model: Mixtral-8x7b-instruct GPU: single A100
Information
Tasks
Reproduction
Steps to reproduce:
Send payload to TGI via http POST request (
/v1/chat/completions
):Receive the following error: {'error': 'EOF while parsing a list at line 1 column 127', 'error_type': 'Input validation error'}
Some more samples and relevant errors:
Stack trace from server:
Expected behavior
Expected behaviour:
Response: {'object': 'chat.completion', 'id': '', 'created': 1721167846, 'model': '/mnt/models', 'system_fingerprint': '2.1.1-sha-4dfdb48', 'choices': [{'index': 0, 'message': {'role': 'assistant', 'tool_calls': [{'id': '0', 'type': 'function', 'function': {'description': None, 'name': 'stdev', 'arguments': {'numbers': [1000, 2000, 3000, 4000, 5000, 7000, 9000, 15000, 20000, 30000]}}}]}, 'logprobs': None, 'finish_reason': 'eos_token'}], 'usage': {'prompt_tokens': 141, 'completion_tokens': 94, 'total_tokens': 235}}
Payload: {'model': 'tgi', 'messages': [{'role': 'user', 'content': "\
[INST] I've been tracking the scoring performance of a certain basketball player across the last 12 games to get insights into his consistency. The points he scored in each game are as follows: 30, 20, 25, 12, 59, 23, 64, 21, 67, 12, 23, and 43. I need to calculate the standard deviation of this scoring to better understand the variability and predictability of his performance. Could you help me with that? [/INST]"}], 'temperature': 0.001, 'tools': [{'type': 'function', 'function': {'name': 'calculate_standard_deviation', 'description': 'Calculates the standard deviation of a list of numbers.', 'parameters': {'type': 'dict', 'properties': {'numbers': {'type': 'array', 'items': {'type': 'number'}, 'description': 'The list of numbers.'}}, 'required': ['numbers']}}}], 'tool_choice': 'calculate_standard_deviation'}Response: {'object': 'chat.completion', 'id': '', 'created': 1721167856, 'model': '/mnt/models', 'system_fingerprint': '2.1.1-sha-4dfdb48', 'choices': [{'index': 0, 'message': {'role': 'assistant', 'tool_calls': [{'id': '0', 'type': 'function', 'function': {'description': None, 'name': 'calculateStandardDeviation', 'arguments': {'numbers': [30, 20, 25, 12, 59, 23, 64, 21, 67, 12, 23, 43]}}}]}, 'logprobs': None, 'finish_reason': 'eos_token'}], 'usage': {'prompt_tokens': 135, 'completion_tokens': 72, 'total_tokens': 207}}
Payload: {'model': 'tgi', 'messages': [{'role': 'user', 'content': "\
[INST] I'm currently tweaking a machine learning model and I need to understand the similarity between two objects in my dataset. Their characteristics are expressed in the feature vectors [0.5, 0.7, 0.2, 0.9, 0.1] for the first object and [0.4, 0.6, 0.3, 0.8, 0.2] for the second one. Could you calculate the cosine similarity between these two feature vectors to help me determine how similar these objects are? [/INST]"}], 'temperature': 0.001, 'tools': [{'type': 'function', 'function': {'name': 'calculate_cosine_similarity', 'description': 'Calculates the cosine similarity of two vectors.', 'parameters': {'type': 'dict', 'properties': {'vectorA': {'type': 'array', 'items': {'type': 'number'}, 'description': 'The first vector.'}, 'vectorB': {'type': 'array', 'items': {'type': 'number'}, 'description': 'The second vector.'}}, 'required': ['vectorA', 'vectorB']}}}], 'tool_choice': 'calculate_cosine_similarity'}Response: {'object': 'chat.completion', 'id': '', 'created': 1721167860, 'model': '/mnt/models', 'system_fingerprint': '2.1.1-sha-4dfdb48', 'choices': [{'index': 0, 'message': {'role': 'assistant', 'tool_calls': [{'id': '0', 'type': 'function', 'function': {'description': None, 'name': 'cosine_similarity', 'arguments': {'vectorA': [0.5, 0.7, 0.2, 0.9, 0.1], 'vectorB': [0.4, 0.6, 0.3, 0.8, 0.2]}}}]}, 'logprobs': None, 'finish_reason': 'eos_token'}], 'usage': {'prompt_tokens': 133, 'completion_tokens': 76, 'total_tokens': 209}}
regex = outlines.fsm.json_schema.build_regex_from_schema(inp_json) interegular.parse_pattern(regex)