search

microsoft / semantic-kernel

Integrate cutting-edge LLM technology quickly and easily into your apps
https://aka.ms/semantic-kernel
MIT License
21.93k stars 3.27k forks source link

.Net: HandlebarsPlanner Run throw [HallucinatedHelpers] The plan references hallucinated helpers: Helper 'substring' #5356

Closed strikene closed 7 months ago

strikene commented 8 months ago

Describe the bug HandlebarsPlanner Run [HallucinatedHelpers] throw The plan references hallucinated helpers: Helper 'substring'

Expected behavior A clear and concise description of what you expected to happen.

Screenshots image

Platform

Additional context here is plan: {{!-- Step 0: Extract key values --}} {{set "subscriptionId" "96000000-0000-0000-0000-a6024fa9bddc"}} {{set "resourceList" "1. DefaultWorkspace-96d73226-adb0-4663-bcfc-a6024fa9bddc-EJP, Japan East; 2. Failure Anomalies - msskapi-d1; 3. Application Insights Smart Detection; 4. msskapi-d1, Japan East, web; 5. ASP-MSSKRSG-823a, Japan East, app; 6. msskapi-d1, Japan East, app; 7. mssksem, Japan East, app; 8. NetworkWatcher_eastasia, East Asia; 9. bs67ase1231sd, Bing.Search.v7; 10. GPT-JPN, Japan East, OpenAI; 11. trans92dgs, Japan West, TextTranslation; 12. dt43bh3tyu876, Japan East; 13. chatmessgaesda, Japan East, StorageV2"}}

{{!-- Step 1: Split the resource list into an array --}} {{set "resourceArray" (array resourceList)}}

{{!-- Step 2: Initialize counters for resource distribution --}} {{set "japanEastCount" 0}} {{set "japanWestCount" 0}} {{set "eastAsiaCount" 0}} {{set "otherCount" 0}}

{{!-- Step 3: Iterate through the resourceArray and count the distribution --}} {{#each resourceArray}} {{#if (equals (substring this -indexOfEnd-11 -indexOfEnd 0) "Japan East")}} {{set "japanEastCount" (add japanEastCount 1)}} {{else if (equals (substring this -indexOfEnd-10 -indexOfEnd 0) "Japan West")}} {{set "japanWestCount" (add japanWestCount 1)}} {{else if (equals (substring this -indexOfEnd-9 -indexOfEnd 0) "East Asia")}} {{set "eastAsiaCount" (add eastAsiaCount 1)}} {{else}} {{set "otherCount" (add otherCount 1)}} {{/if}} {{/each}}

{{!-- Step 4: Output the result --}} {{json (concat "Distribution of resources in Azure Subscription ID " subscriptionId ":" " Japan East: " japanEastCount ", Japan West: " japanWestCount ", East Asia: " eastAsiaCount ", Other regions: " otherCount)}}

Goal : Provide the distribution of resources in the user's Azure subscription with ID 96000000-0000-0000-0000-a6024fa9bddc.

teresaqhoang commented 8 months ago

Hi @strikene, can you share a code snippet of how you're instantiating the kernel and the planner (including the functions / helpers enabled)? Also, how often are you seeing this error?

Thanks

teresaqhoang commented 7 months ago

Closing as stale

strikene commented 7 months ago

Hi @strikene, can you share a code snippet of how you're instantiating the kernel and the planner (including the functions / helpers enabled)? Also, how often are you seeing this error?

Thanks

Here is functions . this._SK_Kernel.ImportPluginFromObject(new HttpPlugin(), nameof(HttpPlugin)); this._SK_Kernel.ImportPluginFromObject(new TextPlugin(), nameof(TextPlugin)); this._SK_Kernel.ImportPluginFromObject(new TimePlugin(), nameof(TimePlugin)); this._SK_Kernel.ImportPluginFromObject(new WaitPlugin(), nameof(WaitPlugin)); this._SK_Kernel.ImportPluginFromObject(new TaskListPlugin(new MicrosoftToDoConnector(graphServiceClient)), "ToDo"); this._SK_Kernel.ImportPluginFromObject(new OrganizationHierarchyPlugin(new OrganizationHierarchyConnector(graphServiceClient)) , "OrganizationHierarchy"); this._SK_Kernel.ImportPluginFromObject(new CalendarPlugin(new OutlookCalendarConnector(graphServiceClient)), "Calendar"); this._SK_Kernel.ImportPluginFromObject(new EmailPlugin(new OutlookMailConnector(graphServiceClient)), "OutLook"); this._SK_Kernel.ImportPluginFromObject(new CloudDrivePlugin(new OneDriveConnector(graphServiceClient)), "OneDrive");

ExtendedAzurePlugin : (This is a function I customized) [Description("Getting an available subscription is the first step in your operation against Azure, " + "The function returns a list containing the display name and SubscriptionId(GUID) of the subscription, respectively ." + "where 'SubscriptionId' is required for subsequent Azure-related operations")] public async Task<IEnumerable> GetAvailableSubscriptions() [Description("Get support tickets that already exist under the given azure subscription ID." + "The function returns a list containing the Title ,Status ,SupportTicketId,SupportEngineer,Severity of the SupportTicket, respectively .")] public async Task<IEnumerable> GetSupportTickets( [Description("Specify the subscription ID(GUID), which is required .you can got ID form 'GetAvailableSubscriptions'")] String SubscriptionId) [Description("Get resource groups that already exist under the given azure subscription ID." + "The function returns a list containing the Name ,Location ,ResourceGroupId,ProvisioningState of the ResourceGroup, respectively .")] public async Task<IEnumerable> GetGetResourceGroup([Description("Specify the subscription ID(GUID), which is required .you can got ID form 'GetAvailableSubscriptions'")] String SubscriptionId)

HandlebarsPlanner selected 'substring' , This is a function that simply does not exist. I often see the HandlebarsPlanner pick such non-existent functions. While the functions we've loaded are already sufficient to complete the task, such as the following task, enumerate the Azure subscription ID, and distribute the resources in my Azure subscription. Although such tasks can sometimes be successful, there is a relatively half chance that the wrong choice will be made.