Planer cannot generate the correct request

[zhongerxin]

In cases where the API for the plugin is complex, Plan cannot generate the correct request. but ChatGPT can be correct, should I use GPT4 ? SCHEMA

bad request

correct request (human edit)

[TaoChenOSU]

Hello @zhongerxin,

Thank you for reporting the issue!

The planner may not always provide the correct inputs to plugins, as the models are indeterministic.

What was the model you were using? The better the model, the more likely you will see correct inputs. But that doesn't necessary mean you have to use GPT4, which is expensive. We have seen good performance from gpt-35-turbo as well.

[zhongerxin]

Hello @zhongerxin,

Thank you for reporting the issue!

The planner may not always provide the correct inputs to plugins, as the models are indeterministic.

What was the model you were using? The better the model, the more likely you will see correct inputs. But that doesn't necessary mean you have to use GPT4, which is expensive. We have seen good performance from gpt-35-turbo as well.

in this case I use gpt3.5-turbo

[TaoChenOSU]

Have you tried with GPT-4? If so, can you let us know what the result was with GPT-4?

We are investigating this issue the planner not providing the correct parameters internally. The planner is known to have the limitation where it cannot handle complex types well due to the lack of function typing.

I will link the issue here later.

[zhongerxin]

@TaoChenOSU Still not work

[TaoChenOSU]

@zhongerxin Thanks for the information! We will take a further look into this.

[TaoChenOSU]

SK epic that could potentially address this issue: https://github.com/orgs/microsoft/projects/866/views/31?pane=issue&itemId=41242062.

Feel free to also play around with the ToManualString function to try to get the type injected into the prompt to help guide the model in constructing the payload better.

[TaoChenOSU]

Closing. Feel free to reopen if you have further questions.