microsoft / TypeChat

TypeChat is a library that makes it easy to build natural language interfaces using types.
https://microsoft.github.io/TypeChat/
MIT License
8.16k stars 390 forks source link

"Restaurant" example has wrong output in README #62

Closed Marcono1234 closed 1 year ago

Marcono1234 commented 1 year ago

The "Restaurant" example has a wrong output in the "Usage" section of the README[^1], if I understand the input correctly, it should be:

-2 large pizza with mushrooms
+1 large pizza with mushrooms
+1 large pizza with sausage
 1 small pizza with sausage
 1 whole Greek salad
 1 Pale Ale
 1 Mack and Jacks

If this is intentional, then maybe there should be a sentence below it explaining this to avoid confusion, for example:

This shows that TypeChat may not be 100% accurate, and you may want to consider asking the user for confirmation before performing any action. The output here erroneously shows 2 mushroom pizzas and 1 sausage pizza, while it should be 1 mushroom pizza and 2 sausage pizzas (one large and one small).

[^1]: And the "Input" might be incorrect as well, shouldn't it be only 🍕> instead of 😀> 🍕>?

DanielRosenwasser commented 1 year ago

@steveluc double-checking to see if this was intentional.

steveluc commented 1 year ago

Looks like just a typo

DanielRosenwasser commented 1 year ago

Open to PRs if you want to fix it before us. 😄

isurumaldeniya commented 1 year ago

can I fix this ? @DanielRosenwasser

weykon commented 1 year ago

Hi ! @isurumaldeniya @DanielRosenwasser I opened a PR #66

weykon commented 1 year ago

the text: I want three pizzas, one with mushrooms and the other two with sausage. Make one sausage a small. And give me a whole Greek and a Pale Ale. And give me a Mack and Jacks.

I ran 5 times, it all output:

    1 large pizza with mushrooms
    2 large pizza with sausage
    1 small pizza with sausage
    1 whole Greek salad
    1 Pale Ale
    1 Mack and Jacks

This shows that TypeChat may not be 100% accurate, and you may want to consider asking the user for confirmation before performing any action. The output here erroneously shows 1 mushroom pizzas and 3 sausage pizza, while it should be 1 mushroom pizza and 2 sausage pizzas (one large and one small).

I think adding this is needed


Before I was using gpt-3.5-turbo, now I am gpt-4-0613, result is correct.

    1 large pizza with mushrooms
    1 small pizza with sausage
    1 large pizza with sausage
    1 whole Greek salad
    1 Pale Ale
    1 Mack and Jacks
steveluc commented 1 year ago

:-) thanks for the diligence on this. From site/src/docs/examples.md (emphasis added):

Name Description
Restaurant An intelligent agent for taking orders at a restaurant. Similar to the coffee shop example, but uses a more complex schema to model more complex linguistic input. The prose files illustrate the line between simpler and more advanced language models in handling compound sentences, distractions, and corrections. This example also shows how we can use TypeScript to provide a user intent summary.
steveluc commented 1 year ago

I think it's pretty amazing that GPT-4 gets examples like this. I agree 100% that we should be clear that not all models will get the examples and also that there is always at least one model that does get the correct output. To work toward that I checked in #67.

steveluc commented 1 year ago

Will leave this open to hear from anyone who wants to suggest further elaboration or clarification.

DanielRosenwasser commented 1 year ago

I also added a few edits to #66 which I think can give some more context. I think that's good to go.