Copilot Chat: Sequential Planner Stablization - Handle missing functions error

[alliscode]

Need to gracefully handle hallucinations from the planner. Options in order of preference:

• Implement retry until we get a valid plan
• Allow missing functions and filter them out (as we were doing before)
• Let the error pass through with a message to user that LLM hallucinated plan

Example of error behavior we're seeing:

Error thrown because LLM keeps proposing functions that are not available "Invalid plan: Failed to find function 'FilterList' in skill 'FilterList'."

Proposed plan:

<plan>\n    
<!-- Get Hypermedia links to resources accessible in GitHub's REST API -->\n    
<function.GitHubSkill.MetaRoot server_url=\"https://api.github.com/\" setContextVariable=\"META_ROOT\"/>\n    \n    
<!-- List details of a pull request by providing its number -->\n    
<function.GitHubSkill.PullsList owner=\"microsoft\" repo=\"semantic-kernel\" state=\"open\" setContextVariable=\"PULL_REQUESTS\"/>\n    \n    
<!-- Filter the pull requests to only show open ones -->\n    
<function.FilterList input=\"$PULL_REQUESTS\" property=\"state\" value=\"open\" setContextVariable=\"OPEN_PULL_REQUESTS\"/>\n    \n    
<!-- Extract the pull request numbers from the open pull requests -->\n    
<function.ExtractValues input=\"$OPEN_PULL_REQUESTS\" property=\"number\" setContextVariable=\"PULL_REQUEST_NUMBERS\"/>\n    \n    
<!-- Loop through the pull request numbers and get the details for each one -->\n    
    <loop>\n        
    <foreach item=\"PULL_REQUEST_NUMBER\" input=\"$PULL_REQUEST_NUMBERS\">\n            
        <!-- Get the details for a single pull request -->\n            
        <function.GitHubSkill.PullsGet owner=\"microsoft\" repo=\"semantic-kernel\" pull_number=\"$PULL_REQUEST_NUMBER\" server_url=\"https://api.github.com/\" setContextVariable=\"PULL_REQUEST_DETAILS\"/>\n            \n            
        <!-- Append the pull request details to the final result -->\n            
        <function.AppendToResult input=\"$PULL_REQUEST_DETAILS\" appendToResult=\"RESULT__PULL_REQUESTS\"/>\n        
        </foreach>\n    
    </loop>\n
</plan>

Skills in context:

[teresaqhoang]

Done https://github.com/microsoft/chat-copilot/pull/87