mudler
commented
1 month ago good catch, this definitely should be handled :+1:
Open vaaale opened 1 month ago
mudler
commented
1 month ago good catch, this definitely should be handled :+1:
LocalAI version: Latest
Environment, CPU architecture, OS, and Version: Not relevant
Describe the bug This bug has two elements to it. The first is that the server segfaults if it receives an invalid JSON schema. The second element is that only the 2019-09 JSON schema format is supported but not the 06,07 version. https://opis.io/json-schema/2.x/definitions.html The relevant difference is that the keyword "definitions" was changed to "$defs" in the 2019-09 version. Unfortunately, Python libraries such as Pydantic version 1 use the 06,07 version, which libraries such as LlamaIndex use for function calling. (Unfortunately). The crash happens when the resolveReference function in grammar_json_schema.go fails to find #/$defs/ in the schema. (See below)
To Reproduce To reproduce you can either run the LlamaIndex QueryPlanner example from here: https://docs.llamaindex.ai/en/stable/examples/agent/openai_agent_query_plan/
Or execute the following 'curl' request:
curl -X POST -H "Content-Type: application/json" -d @msg.json http://localhost:5000/v1/chat/completionsWhere msg.json has the following content: (JSON file [attached) msg.jsonExpected behavior
Logs
10:59AM DBG Request received: {"model":"gpt-3.5-turbo","language":"","translate":false,"n":0,"top_p":null,"top_k":null,"temperature":0.1,"max_tokens":null,"echo":false,"batch":0,"ignore_eos":false,"repeat_penalty":0,"repeat_last_n":0,"n_keep":0,"frequency_penalty":0,"presence_penalty":0,"tfz":null,"typical_p":null,"seed":null,"negative_prompt":"","rope_freq_base":0,"rope_freq_scale":0,"negative_prompt_scale":0,"use_fast_tokenizer":false,"clip_skip":0,"tokenizer":"","file":"","size":"","prompt":null,"instruction":"","input":null,"stop":null,"messages":[{"role":"user","content":"What were the risk factors in sept 2022?"}],"functions":null,"function_call":null,"tools":[{"type":"function","function":{"name":"query_plan_tool","description":" This is a query plan tool that takes in a list of tools and executes a query plan over these tools to answer a query. The query plan is a DAG of query nodes.\n\nGiven a list of tool names and the query plan schema, you can choose to generate a query plan to answer a question.\n\nThe tool names and descriptions are as follows:\n\n\n\n Tool Name: sept_2022\nTool Description: Provides information about Uber quarterly financials ending September 2022 \n\nTool Name: june_2022\nTool Description: Provides information about Uber quarterly financials ending June 2022 \n\nTool Name: march_2022\nTool Description: Provides information about Uber quarterly financials ending March 2022 \n ","parameters":{"definitions":{"QueryNode":{"description":"Query node.\n\nA query node represents a query (query_str) that must be answered.\nIt can either be answered by a tool (tool_name), or by a list of child nodes\n(child_nodes).\nThe tool_name and child_nodes fields are mutually exclusive.","properties":{"dependencies":{"description":"List of sub-questions that need to be answered in order to answer the question given byquery_str.Should be blank if there are no sub-questions to be specified, in which casetool_nameis specified.","items":{"type":"integer"},"title":"Dependencies","type":"array"},"id":{"description":"ID of the query node.","title":"Id","type":"integer"},"query_str":{"description":"Question we are asking. This is the query string that will be executed. ","title":"Query Str","type":"string"},"tool_name":{"description":"Name of the tool to execute thequery_str`.","title":"Tool Name","type":"string"}},"required":["id","query_str"],"title":"QueryNode","type":"object"}},"properties":{"nodes":{"description":"The original question we are asking.","items":{"$ref":"#/definitions/QueryNode"},"title":"Nodes","type":"array"}},"required":["nodes"],"type":"object"}}}],"tool_choice":"auto","stream":false,"mode":0,"step":0,"grammar":"","grammar_json_functions":null,"grammar_json_name":null,"backend":"","model_base_name":""} 10:59AM DBG guessDefaultsFromFile: template already set name=gpt-3.5-turbo 10:59AM DBG Configuration read: &{PredictionOptions:{Model:00c61256fb64047d5f0ffecaaffec50a Language: Translate:false N:0 TopP:0xc00003c7a0 TopK:0xc00003c7a8 Temperature:0xc000712e00 Maxtokens:0xc00003c7e0 Echo:false Batch:0 IgnoreEOS:false RepeatPenalty:0 RepeatLastN:0 Keep:0 FrequencyPenalty:0 PresencePenalty:0 TFZ:0xc00003c7d8 TypicalP:0xc00003c7d0 Seed:0xc00003c7f8 NegativePrompt: RopeFreqBase:0 RopeFreqScale:0 NegativePromptScale:0 UseFastTokenizer:false ClipSkip:0 Tokenizer:} Name:gpt-3.5-turbo F16:0xc00003c780 Threads:0xc00003c790 Debug:0xc000713140 Roles:map[] Embeddings:0xc00003c7f1 Backend: TemplateConfig:{Chat:{{.Input -}} <|im_start|>assistant ChatMessage:<|im_start|>{{if eq .RoleName "assistant"}}assistant{{else if eq .RoleName "system"}}system{{else if eq .RoleName "tool"}}tool{{else if eq .RoleName "user"}}user{{end}} {{- if .FunctionCall }}{{- else if eq .RoleName "tool" }} {{- end }}<|im_end|> Completion:{{.Input}} Edit: Functions:<|im_start|>system You are a function calling AI model. You are provided with function signatures within{{range .Functions}}{'type': 'function', 'function': {'name': '{{.Name}}', 'description': '{{.Description}}', 'parameters': {{toJson .Parameters}}}}{{end}}
Use the following pydantic model json schema for each tool call you will make:
{'title': 'FunctionCall', 'type': 'object', 'properties': {'arguments': {'title': 'Arguments', 'type': 'object'}, 'name': {'title': 'Name', 'type': 'string'}}, 'required': ['arguments', 'name']}
Use the 'answer' function when you are ready to write your final answer to the users request.
For each function call return a json object with function name and arguments within