Open jimmerzzz52 opened 1 week ago
jimmerzzz52
commented
1 week ago We have been able to isolate the issue to the keys in the curl request above. We need to remove the dash "-" from each key that is generated.
e.g. FredStructuredSearchPlugin-FredSearchAsynch needs to be renamed FredStructuredSearchPlugin_FredSearchAsynch or something similar.
jimmerzzz52
commented
1 week ago Hi there. Here is what we have, we have created a custom plugin by extending the search plugin provided by AISearch
Below is the code:
from plugins.base.AISearchPlugin import AISearchPlugin
from semantic_kernel.functions.kernel_function_decorator import kernel_function
from typing_extensions import Annotated
class ExxonSearchPlugin:
"""
An AI Search engine skill for the Exxon Index.
"""
def __init__(self, history) -> None:
self._history = history
@kernel_function(
description='Performs a knowledgebase search for a given query and responds with the the answer and a citation if it is found.The index contains data about Exxon from historic 10-K filings from 2015 through 2023.Since 10-K filings are filed at the end of the year, the search does not include data from the current year.',
name="ExxonSearchAsync",
)
async def search_async(self, query: Annotated[str, "The search query"]) -> str:
return await AISearchPlugin.search_async(
self, query, "exxon-community-based-index"
)
// SKPlanner code / config
import asyncio
import datetime
import json
import logging
import os
import semantic_kernel as sk
from jinja2 import Environment, FileSystemLoader
from plugins.native.JnJSearchPlugin import JnJSearchPlugin
from plugins.native.ExxonSearchPlugin import ExxonSearchPlugin
from plugins.base.FredStructuredSearchPlugin import FredStructuredSearchPlugin
from plugins.native.math_plugin import MathPlugin
from plugins.native.ExxonDBPlugin import ExxonDBPlugin
from promptflow.core import tool
from semantic_kernel.connectors.ai.open_ai import (
AzureChatCompletion,
OpenAIChatPromptExecutionSettings,
)
from semantic_kernel.core_plugins import TextPlugin, TimePlugin
from semantic_kernel.planners.function_calling_stepwise_planner import (
FunctionCallingStepwisePlanner,
FunctionCallingStepwisePlannerOptions,
)
from utils.auth import Auth
from utils.oai import render_with_token_limit
from utils.chain_of_thought import simplifyChainOfThought
@tool
async def skplanner_tool(
question: str,
history: list,
):
kernel = sk.Kernel(log=logging.getLogger("planner"))
service_id = "default"
kernel.add_service(
AzureChatCompletion(
service_id=service_id,
deployment_name=os.environ.get("AZURE_OPENAI_DEPLOYMENT_NAME"),
endpoint=os.environ.get("OPENAI_API_BASE"),
api_key=Auth.get_openai_key(),
api_version=os.environ["OPENAI_API_VERSION"],
)
)
qna_template = Environment(
loader=FileSystemLoader(os.path.dirname(os.path.abspath(__file__)))
).get_template("prompts/qna_prompt_with_metadata.md")
qna_prompt = render_with_token_limit(
qna_template,
int(os.environ["PROMPT_TOKEN_LIMIT"]),
question=question,
date=datetime.date.today(),
history=history,
)
kernel.add_plugin(plugin_name="math", plugin=MathPlugin())
kernel.add_plugin(plugin_name="time", plugin=TimePlugin())
kernel.add_plugin(plugin_name="text", plugin=TextPlugin())
kernel.add_plugin(ExxonSearchPlugin(history), plugin_name="ExxonSearch")
kernel.add_plugin(
FredStructuredSearchPlugin(history), plugin_name="FredStructuredSearchPlugin"
)
kernel.add_plugin(JnJSearchPlugin(history), plugin_name="JohnsonAndJohnsonSearch")
execution_settings = OpenAIChatPromptExecutionSettings(
service_id=service_id, temperature=0, max_tokens=4000
)
planner = FunctionCallingStepwisePlanner(
service_id=service_id,
options=FunctionCallingStepwisePlannerOptions(
max_iterations=50,
min_iteration_time_ms=1000,
execution_settings=execution_settings,
),
)
result = await planner.invoke(kernel=kernel, question=qna_prompt)
chain_of_thought = simplifyChainOfThought(result.chat_history.model_dump_json())
# print(chain_of_thought)
return_value = {
"answer": result.final_answer,
"chain_of_thought": chain_of_thought or "",
}
return return_value
markwallace-microsoft
commented
1 week ago We have been able to isolate the issue to the keys in the curl request above. We need to remove the dash "-" from each key that is generated.
e.g. FredStructuredSearchPlugin-FredSearchAsynch needs to be renamed FredStructuredSearchPlugin_FredSearchAsynch or something similar.
Hi @jimmerzzz52 using the hash character in the function name is valid, here is the relevant section from the API Reference - OpenAI API:
name
string
Required
The name of the function to be called. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.I can reproduce the issue when using gpt-4o, the 500 Internal Server Error response indicates this is an OpenAI issue. I tried the same request using gpt-4 and I get the following response, so a short term solution could be to switch to that model.
{
"id": "chatcmpl-9g7ZwV8KhLrcGtuXDbcCxrDpvDOWg",
"object": "chat.completion",
"created": 1719825936,
"model": "gpt-4-0613",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": null,
"tool_calls": [
{
"id": "call_T5x7SKXu1tG4VwhbfPEb3nRR",
"type": "function",
"function": {
"name": "FredStructuredSearchPlugin-FredSearchAsynch",
"arguments": "{\"query\":\"WTI crude oil from 2023-06-29 to 2024-06-28\"}"
}
}
]
},
"logprobs": null,
"finish_reason": "tool_calls"
}
],
"usage": {
"prompt_tokens": 1588,
"completion_tokens": 38,
"total_tokens": 1626
},
"system_fingerprint": null
}Thank you Mark for testing. It also looks like other request keys have dashes in them, so it does not make sense why Open AI is erroring on just one key. Do we have a contact on the Open AI team who could help?
markwallace-microsoft
commented
1 week ago Consider allowing the delimiter to be configured to support non OpenAI models
eavanvalkenburg
commented
6 days ago @jimmerzzz52 have you also tried to do this with plain function calling, we've found that it is often easier and better then using the planners for this, this sample has that and we've built in in such a way that if the model sends back multiple tool calls then those get executed in paralel and then it goes back to the model for either a answer or another set of tool calls, so in that way it is able to do more complex tasks, similar to a planner.
Describe the bug Semantic Kernel Errors when a complex prompt is inputting with the 4o model. A 500 is given from Open AI.
To Reproduce Steps to reproduce the behavior:
Example Curl: Plugin names/key in json have dashes which is causing the call to fail curl --location 'https://oai-kaia-demo.openai.azure.com/openai/deployments/oai-gpt-4o/chat/completions?api-version=2024-02-01' \ --header 'host: oai-kaia-demo.openai.azure.com' \ --header 'accept-encoding: gzip, deflate' \ --header 'connection: keep-alive' \ --header 'accept: application/json' \ --header 'content-type: application/json' \ --header 'user-agent: Semantic-Kernel' \ --header 'x-stainless-lang: python' \ --header 'x-stainless-package-version: 1.34.0' \ --header 'x-stainless-os: MacOS' \ --header 'x-stainless-arch: arm64' \ --header 'x-stainless-runtime: CPython' \ --header 'x-stainless-runtime-version: 3.12.3' \ --header 'semantic-kernel-version: python-1.1.1' \ --header 'x-stainless-async: async:asyncio' \ --header 'x-ms-useragent: promptflow-cli/1.12.0 promptflow-sdk/1.12.0 promptflow/1.12.0 promptflow-core/1.12.0 promptflow-tracing/1.12.0' \ --header 'ms-azure-ai-promptflow: {"execution_target": "dag", "run_mode": "Test", "flow_id": "default_flow_id", "root_run_id": "dc164f8d-ed4f-485a-935f-eda7e331b202"}' \ --header 'content-length: 11222' \ --header 'x-postman-captr: 4090185' \ --data '{"messages":[{"role":"user","content":"Original request: # Current Date\nThe current date is 2024-06-28\n\n# Instructions\nALL responses should be in English\n\nYou are a helpful intelligent analyst who can answer the question at the end based on context available to you via your included plugins as well as the previous conversation history between you and human. Ensure that when you answer you are concise and show your work.\n\nTry as much as you can to answer based on the the context available in your plugins even if you have to use the plugins more than once to answer the question. Always use math plugins when doing trend analysis to ensure accuracy. If you cannot derive the answer from the context, you should say you don'\''t know.\n\n# Conversation history\n\n\n# Question\nProvide data from WTI crude oil and compare it to the S&P 500 over the past year.\n\nYou are in the process of helping the user fulfill this request using the following plan:\nTo provide data from WTI crude oil and compare it to the S&P 500 over the past year, we need to follow these steps:\n\n1. Get the current date: We already know the current date is 2024-06-28.\n2. Calculate the date one year ago: This will be 2023-06-28.\n3. Search for WTI crude oil data from 2023-06-28 to 2024-06-28.\n4. Search for S&P 500 data from 2023-06-28 to 2024-06-28.\n5. Compare the trends of WTI crude oil and S&P 500 over the past year.\n\nLet'\''s proceed step by step.\n\n### Step 1: Calculate the date one year ago\nWe need to calculate the date one year ago from 2024-06-28.\n\n### Step 2: Search for WTI crude oil data from 2023-06-28 to 2024-06-28\nWe will use the FredStructuredSearchPlugin to search for WTI crude oil data.\n\n### Step 3: Search for S&P 500 data from 2023-06-28 to\n\nThe user will ask you for help with each step."},{"role":"user","content":"Perform the next step of the plan if there is more work to do. When you have reached a final answer, use the UserInteraction-SendFinalAnswer function to communicate this back to the user."},{"role":"assistant","tool_calls":[{"id":"call_UNdlBmcKm0CnFHxX7D9MGvBf","type":"function","function":{"name":"time-days_ago","arguments":"{\"days\":\"365\"}"}}]},{"role":"tool","content":"Thursday, 29 June, 2023","tool_call_id":"call_UNdlBmcKm0CnFHxX7D9MGvBf"},{"role":"user","content":"Perform the next step of the plan if there is more work to do. When you have reached a final answer, use the UserInteraction-SendFinalAnswer function to communicate this back to the user."}],"model":"oai-gpt-4o","max_tokens":4000,"stream":false,"temperature":0,"tool_choice":"auto","tools":[{"type":"function","function":{"name":"math-Add","description":"Returns the Addition result of the values provided.","parameters":{"type":"object","properties":{"input":{"type":"integer","description":"the first number to add"},"amount":{"type":"integer","description":"the second number to add"}},"required":["input","amount"]}}},{"type":"function","function":{"name":"math-Divide","description":"Divides value by a value","parameters":{"type":"object","properties":{"input":{"type":"integer","description":"the first number"},"amount":{"type":"integer","description":"the number to divide by"}},"required":["input","amount"]}}},{"type":"function","function":{"name":"math-Multiply","description":"Returns the Multiplication result of the values provided.","parameters":{"type":"object","properties":{"input":{"type":"integer","description":"the first number to multiply"},"amount":{"type":"integer","description":"the second number to multiply"}},"required":["input","amount"]}}},{"type":"function","function":{"name":"math-Subtract","description":"Subtracts value to a value","parameters":{"type":"object","properties":{"input":{"type":"integer","description":"the first number"},"amount":{"type":"integer","description":"the number to subtract"}},"required":["input","amount"]}}},{"type":"function","function":{"name":"time-date","description":"Get the current date.","parameters":{"type":"object","properties":{},"required":[]}}},{"type":"function","function":{"name":"time-date_matching_last_day_name","description":"Get the date of the last day matching the supplied week day name in English.","parameters":{"type":"object","properties":{"day_name":{"type":"string"}},"required":["day_name"]}}},{"type":"function","function":{"name":"time-day","description":"Get the current day","parameters":{"type":"object","properties":{},"required":[]}}},{"type":"function","function":{"name":"time-dayOfWeek","description":"Get the current day of the week","parameters":{"type":"object","properties":{},"required":[]}}},{"type":"function","function":{"name":"time-days_ago","description":"Get the date of offset from today by a provided number of days","parameters":{"type":"object","properties":{"days":{"type":"string"}},"required":["days"]}}},{"type":"function","function":{"name":"time-hour","description":"Get the current hour","parameters":{"type":"object","properties":{},"required":[]}}},{"type":"function","function":{"name":"time-hourNumber","description":"Get the current hour number","parameters":{"type":"object","properties":{},"required":[]}}},{"type":"function","function":{"name":"time-iso_date","description":"Get the current date in iso format.","parameters":{"type":"object","properties":{},"required":[]}}},{"type":"function","function":{"name":"time-minute","description":"Get the current minute","parameters":{"type":"object","properties":{},"required":[]}}},{"type":"function","function":{"name":"time-month","description":"Get the current month","parameters":{"type":"object","properties":{},"required":[]}}},{"type":"function","function":{"name":"time-month_number","description":"Get the current month number","parameters":{"type":"object","properties":{},"required":[]}}},{"type":"function","function":{"name":"time-now","description":"Get the current date and time in the local time zone","parameters":{"type":"object","properties":{},"required":[]}}},{"type":"function","function":{"name":"time-second","description":"Get the seconds on the current minute","parameters":{"type":"object","properties":{},"required":[]}}},{"type":"function","function":{"name":"time-time","description":"Get the current time in the local time zone","parameters":{"type":"object","properties":{},"required":[]}}},{"type":"function","function":{"name":"time-timeZoneName","description":"Get the current time zone name","parameters":{"type":"object","properties":{},"required":[]}}},{"type":"function","function":{"name":"time-timeZoneOffset","description":"Get the current time zone offset","parameters":{"type":"object","properties":{},"required":[]}}},{"type":"function","function":{"name":"time-today","description":"Get the current date.","parameters":{"type":"object","properties":{},"required":[]}}},{"type":"function","function":{"name":"time-utcNow","description":"Get the current date and time in UTC","parameters":{"type":"object","properties":{},"required":[]}}},{"type":"function","function":{"name":"time-year","description":"Get the current year","parameters":{"type":"object","properties":{},"required":[]}}},{"type":"function","function":{"name":"text-lowercase","description":"Convert a string to lowercase.","parameters":{"type":"object","properties":{"input":{"type":"string"}},"required":["input"]}}},{"type":"function","function":{"name":"text-trim","description":"Trim whitespace from the start and end of a string.","parameters":{"type":"object","properties":{"input":{"type":"string"}},"required":["input"]}}},{"type":"function","function":{"name":"text-trim_end","description":"Trim whitespace from the end of a string.","parameters":{"type":"object","properties":{"input":{"type":"string"}},"required":["input"]}}},{"type":"function","function":{"name":"text-trim_start","description":"Trim whitespace from the start of a string.","parameters":{"type":"object","properties":{"input":{"type":"string"}},"required":["input"]}}},{"type":"function","function":{"name":"text-uppercase","description":"Convert a string to uppercase.","parameters":{"type":"object","properties":{"input":{"type":"string"}},"required":["input"]}}},{"type":"function","function":{"name":"ExxonSearch-ExxonSearchAsync","description":"Performs a knowledgebase search for a given query and responds with the the answer and a citation if it is found. \n The index contains data about Exxon from historic 10-K filings from 2015 through 2023. \n Since 10-K filings are filed at the end of the year, the search does not include data from the current year.","parameters":{"type":"object","properties":{"query":{"type":"string","description":"The search query"}},"required":["query"]}}},{"type":"function","function":{"name":"FredStructuredSearchPlugin-FredSearchAsynch","description":"\n Performs a targeted search on economic data and responds with the results of a \n query from Fred'\''s API. Fred contains frequently updated US macro and regional \n economic time series. We want to reference this data if it is relevant to any of the \n prompts asked before. That way we can look into the data for more contextual information.\n ","parameters":{"type":"object","properties":{"query":{"type":"string","description":"The search query"}},"required":["query"]}}},{"type":"function","function":{"name":"JohnsonAndJohnsonSearch-JnJsearchAsync","description":"Performs a knowledgebase search for a given query and responds with the the answer and a citation if it is found. \n The index contains data about Johnson and Johnson from historic 10-K filings from 2017 through 2022. \n Since 10-K filings are filed at the end of the year, the search does not include data from the current year.","parameters":{"type":"object","properties":{"query":{"type":"string","description":"The search query"}},"required":["query"]}}},{"type":"function","function":{"name":"UserInteraction-SendFinalAnswer","description":"The final answer to return to the user","parameters":{"type":"object","properties":{"answer":{"type":"string","description":"The final answer"}},"required":["answer"]}}},{"type":"function","function":{"name":"sequential_planner-create_plan","description":"Create a plan for the given goal","parameters":{"type":"object","properties":{"available_functions":{"type":"object","description":"A list of functions that can be used to generate the plan"},"goal":{"type":"object","description":"The goal to satisfy"},"name_delimiter":{"type":"object"}},"required":["available_functions","goal","name_delimiter"]}}}]}'
Expected behavior A successful response instead of a 500 response from OpenAI
Platform
Additional context We have tried may different ways to make the function call work, but we are dealing with issues.