search

microsoft / semantic-kernel

Integrate cutting-edge LLM technology quickly and easily into your apps
https://aka.ms/semantic-kernel
MIT License
21.97k stars 3.27k forks source link

Plan with imported skills from ChatGPT plugins , got unexpected results #661

Closed sheng-jie closed 1 year ago

sheng-jie commented 1 year ago

I modified Example21_ChatGPTPlugins.cs in local to run with planner, got unexpected results.

var kernel = new KernelBuilder().WithLogger(ConsoleLogger.Log).Build();

kernel.Config.AddAzureTextCompletionService(
            Env.Var("AZURE_OPENAI_SERVICE_ID"),
            Env.Var("AZURE_OPENAI_DEPLOYMENT_NAME"),
            Env.Var("AZURE_OPENAI_ENDPOINT"),
            Env.Var("AZURE_OPENAI_KEY"));

var skill = await kernel.ImportChatGptPluginSkillFromUrlAsync("Klarna", new Uri("https://www.klarna.com/.well-known/ai-plugin.json"));

var contextVariables = new ContextVariables();
contextVariables.Set("q", "Laptop");     //A precise query that matches one very small category or product that needs to be searched for to find the products the user is looking for. If the user 
explicitly stated what they want, use that as a query. The query is as specific as possible to the product name or category mentioned by the user in its singular form, and don't contain any 
clarifiers like latest, newest, cheapest, budget, premium, expensive or similar. The query is always taken from the latest topic, if there is a new topic a new query is started.
contextVariables.Set("size", "3");        // number of products returned
contextVariables.Set("budget", "200");    // maximum price of the matching product in local currency, filters results
var rst = await kernel.RunAsync(contextVariables, skill["productsUsingGET"]);
Console.WriteLine("skill response: {0}", rst);

//run with planner
var planner = new SequentialPlanner(kernel, new PlannerConfig() { MaxTokens = 1024 });
var originalPlan = await planner.CreatePlanAsync("I want to buy a laptop and my budget is 200 dollars");
Console.WriteLine("Original plan:");
Console.WriteLine(originalPlan.ToJson());
var result = await kernel.RunAsync(originalPlan);
Console.WriteLine("plan response: {0}", result);
Console.ReadLine();

The results as follows:

skill response: {
  "content": "{\u0022products\u0022:[{\u0022name\u0022:\u0022Apple MacBook Air (2020) M1 OC 7C GPU 8GB 256GB SSD 13\\\u0022\u0022,\u0022url\u0022:\u0022https://www.klarna.com/us/shopping/pl/cl27/3200009205/Laptops/Apple-MacBook-Air-%282020%29-M1-OC-7C-GPU-8GB-256GB-SSD-13/?utm_source=openai\u0026ref-site=openai_plugin\u0022,\u0022price\u0022:\u0022$799.99\u0022,\u0022attributes\u0022:[\u0022Processor:Apple M1\u0022,\u0022Hard Drive Type:SSD\u0022,\u0022Screen Size:13.3\\\u0022\u0022,\u0022Operating System:macOS\u0022,\u0022Series:Apple Macbook Air\u0022,\u0022SSD Size:256 GB\u0022,\u0022RAM:8 GB\u0022,\u0022Release Year:2020\u0022]},{\u0022name\u0022:\u0022Apple MacBook Air (2022) M2 OC 8C GPU 8GB 256GB SSD 13.6\\\u0022\u0022,\u0022url\u0022:\u0022https://www.klarna.com/us/shopping/pl/cl27/3201585850/Laptops/Apple-MacBook-Air-%282022%29-M2-OC-8C-GPU-8GB-256GB-SSD-13.6/?utm_source=openai\u0026ref-site=openai_plugin\u0022,\u0022price\u0022:\u0022$1049.00\u0022,\u0022attributes\u0022:[\u0022Processor:Apple M2\u0022,\u0022Hard Drive Type:SSD\u0022,\u0022Screen Size:13.6\\\u0022\u0022,\u0022Operating System:macOS\u0022,\u0022Series:Apple Macbook Air\u0022,\u0022SSD Size:256 GB\u0022,\u0022RAM:8 GB\u0022,\u0022Release Year:2022\u0022]},{\u0022name\u0022:\u0022HP 14-dq0055dx\u0022,\u0022url\u0022:\u0022https://www.klarna.com/us/shopping/pl/cl27/3202767894/Laptops/HP-14-dq0055dx/?utm_source=openai\u0026ref-site=openai_plugin\u0022,\u0022price\u0022:\u0022$179.99\u0022,\u0022attributes\u0022:[\u0022Processor:Intel Celeron\u0022,\u0022Hard Drive Type:SSD\u0022,\u0022Screen Size:14.0\\\u0022\u0022,\u0022Windows Version:Windows 11 Home\u0022,\u0022Operating System:Windows\u0022,\u0022SSD Size:64 GB\u0022,\u0022RAM:4 GB\u0022,\u0022Release Year:2022\u0022]}]}",
  "contentType": "application/json; charset=utf-8"
}
Original plan:
{"state":[{"Key":"INPUT","Value":""}],"steps":[{"state":[{"Key":"INPUT","Value":""}],"steps":[],"named_parameters":[{"Key":"max_price","Value":"200"},{"Key":"size","Value":"10"},{"Key":"server_url","Value":"https://example.com"},{"Key":"q","Value":"laptop"},{"Key":"min_price","Value":"0"},{"Key":"INPUT","Value":""}],"named_outputs":[{"Key":"PRODUCTS","Value":""},{"Key":"INPUT","Value":""}],"next_step_index":0,"name":"productsUsingGET","skill_name":"Klarna","description":null},{"state":[{"Key":"INPUT","Value":""}],"steps":[],"named_parameters":[{"Key":"INPUT","Value":""}],"named_outputs":[{"Key":"INPUT","Value":""}],"next_step_index":0,"name":"","skill_name":"Microsoft.SemanticKernel.Orchestration.Plan","description":""}],"named_parameters":[{"Key":"INPUT","Value":""}],"named_outputs":[{"Key":"INPUT","Value":""}],"next_step_index":0,"name":"I want to buy a laptop and my budget is 200 dollars","skill_name":"Microsoft.SemanticKernel.Orchestration.Plan","description":"I want to buy a laptop and my budget is 200 dollars"}
warn: object[0]
      Something went wrong while rendering the Rest function. Function: Klarna.productsUsingGET. Error: Response status code does not indicate success: 404 (Not Found).
      System.Net.Http.HttpRequestException: Response status code does not indicate success: 404 (Not Found).
         at System.Net.Http.HttpResponseMessage.EnsureSuccessStatusCode()
         at Microsoft.SemanticKernel.Connectors.WebApi.Rest.RestApiOperationRunner.SendAsync(Uri url, HttpMethod method, IDictionary`2 headers, HttpContent payload, CancellationToken cancellationToken) in D:\OpenAI\semantic-kernel\dotnet\src\SemanticKernel.Abstractions\Connectors\WebApi\Rest\RestApiOperationRunner.cs:line 97
         at Microsoft.SemanticKernel.KernelOpenApiExtensions.<>c__DisplayClass5_0.<<RegisterRestApiFunction>g__ExecuteAsync|0>d.MoveNext() in D:\OpenAI\semantic-kernel\dotnet\src\Skills\Skills.OpenAPI\Extensions\KernelOpenApiExtensions.cs:line 280

We can see that the generated plan is wrong, the server_url is not correct, it should be https://www.klarna.com/us/shopping.

{
    "state": [
        {
            "Key": "INPUT",
            "Value": ""
        }
    ],
    "steps": [
        {
            "state": [
                {
                    "Key": "INPUT",
                    "Value": ""
                }
            ],
            "steps": [],
            "named_parameters": [
                {
                    "Key": "max_price",
                    "Value": "200"
                },
                {
                    "Key": "size",
                    "Value": "10"
                },
                {
                    "Key": "server_url",
                    "Value": "https://example.com"
                },
                {
                    "Key": "q",
                    "Value": "laptop"
                },
                {
                    "Key": "min_price",
                    "Value": "0"
                },
                {
                    "Key": "INPUT",
                    "Value": ""
                }
            ],
            "named_outputs": [
                {
                    "Key": "PRODUCTS",
                    "Value": ""
                },
                {
                    "Key": "INPUT",
                    "Value": ""
                }
            ],
            "next_step_index": 0,
            "name": "productsUsingGET",
            "skill_name": "Klarna",
            "description": null
        },
        {
            "state": [
                {
                    "Key": "INPUT",
                    "Value": ""
                }
            ],
            "steps": [],
            "named_parameters": [
                {
                    "Key": "INPUT",
                    "Value": ""
                }
            ],
            "named_outputs": [
                {
                    "Key": "INPUT",
                    "Value": ""
                }
            ],
            "next_step_index": 0,
            "name": "",
            "skill_name": "Microsoft.SemanticKernel.Orchestration.Plan",
            "description": ""
        }
    ],
    "named_parameters": [
        {
            "Key": "INPUT",
            "Value": ""
        }
    ],
    "named_outputs": [
        {
            "Key": "INPUT",
            "Value": ""
        }
    ],
    "next_step_index": 0,
    "name": "I want to buy a laptop and my budget is 200 dollars",
    "skill_name": "Microsoft.SemanticKernel.Orchestration.Plan",
    "description": "I want to buy a laptop and my budget is 200 dollars"
}
reconsumeralization commented 1 year ago

To fix the issue with the incorrect server URL, you need to update the planner.CreatePlanAsync method call with the correct URL for the Klarna skill. Here's the updated code:

var originalPlan = await planner.CreatePlanAsync("I want to buy a laptop and my budget is 200 dollars", skill["productsUsingGET"]); Make sure to include the skill["productsUsingGET"] parameter in the planner.CreatePlanAsync method call to specify the skill to be used for the plan.

Additionally, you need to update the server_url named parameter in the named_parameters list of the first step in the plan to use the correct Klarna server URL. Here's the updated code snippet:

var originalPlan = await planner.CreatePlanAsync("I want to buy a laptop and my budget is 200 dollars"); originalPlan.Steps[0].NamedParameters.First(p => p.Key == "server_url").Value = "https://www.klarna.com/us/shopping"; After these updates, the code should execute correctly, and the plan and response should reflect the desired behavior.