microsoft / TypeChat

TypeChat is a library that makes it easy to build natural language interfaces using types.
https://microsoft.github.io/TypeChat/
MIT License
8.06k stars 378 forks source link

New (Python) example: simple drawings (rectangle, ellipse, arrow) #238

Open gvanrossum opened 2 months ago

gvanrossum commented 2 months ago

For me, this works best with openai model gpt-3.5-turbo; gpt-4 seems worse (?!).

Here's a sample session (no longer valid without history support) ``` PATH=/opt/homebrew/opt/ccache/libexec:/opt/homebrew/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/local/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/bin:/var/run/com.apple.security.cryptexd/codex.system/bootstrap/usr/appleinternal/bin:/Library/Apple/usr/bin:/usr/local/share/dotnet:~/.dotnet/tools:/Users/guido/.vscode/extensions/ms-python.python-2024.4.1/python_files/deactivate/bash:/Users/guido/TypeChat/python/.venv/bin:/Library/Frameworks/Python.framework/Versions/3.13/bin:/Library/Frameworks/Python.framework/Versions/3.11/bin:/Library/Frameworks/Python.framework/Versions/3.12/bin:/Users/guido/.nvm/versions/node/v19.8.1/bin:/opt/homebrew/opt/ccache/libexec:/opt/homebrew/bin The default interactive shell is now zsh. To update your account to use zsh, please run `chsh -s /bin/zsh`. For more details, please visit https://support.apple.com/kb/HT208050. (.venv) ~/TypeChat/python$ PYTHONPATH=src python examples/drawing/demo.py ~> draw three squares of side 50 in a diagonal --------- NEXT REQUEST --------- {'role': 'user', 'content': '\nYou are a service that translates user requests into JSON objects of type "Drawing" according to the following TypeScript definitions:\n```\n// A drawing is a list of boxes. (We\'ll add other elements later, like arrows.)\ninterface Drawing {\n    items: Array;\n}\n\n// Use this type for input that match nothing else\ninterface UnknownText {\n    type: "Unknown";\n    // The text that wasn\'t understood\n    text: string;\n}\n\n// A rectangular box.\n// \n// The coordinate system has origin top left, x points right, y points down.\n// Measurements are in pixels.\n// \n// There can also be text in the box. There are optional style properties.\ninterface Box {\n    type: "Box";\n    // Top left corner coordinates\n    x: number;\n    y: number;\n    // Size of the box\n    width: number;\n    height: number;\n    // Text centered in the box\n    text: string;\n    // Box drawing style (optional)\n    style: Style | null;\n}\n\ninterface Style {\n    type: "Style";\n    corners: "rounded" | "sharp";\n}\n\n```\n'} {'role': 'user', 'content': "\nThe following is a user request:\n'''\ndraw three squares of side 50 in a diagonal\n'''\n\nThe following is the user request translated into a JSON object with 2 spaces of indentation and no properties with the value undefined:\n"} retrying (ReadTimeout('')) ... retrying (ReadTimeout('')) ... retrying (ReadTimeout('')) ... ReadTimeout('') raised from within internal TypeChat language model. ~> draw two squares of side 50 in a diagonal --------- NEXT REQUEST --------- {'role': 'user', 'content': '\nYou are a service that translates user requests into JSON objects of type "Drawing" according to the following TypeScript definitions:\n```\n// A drawing is a list of boxes. (We\'ll add other elements later, like arrows.)\ninterface Drawing {\n    items: Array;\n}\n\n// Use this type for input that match nothing else\ninterface UnknownText {\n    type: "Unknown";\n    // The text that wasn\'t understood\n    text: string;\n}\n\n// A rectangular box.\n// \n// The coordinate system has origin top left, x points right, y points down.\n// Measurements are in pixels.\n// \n// There can also be text in the box. There are optional style properties.\ninterface Box {\n    type: "Box";\n    // Top left corner coordinates\n    x: number;\n    y: number;\n    // Size of the box\n    width: number;\n    height: number;\n    // Text centered in the box\n    text: string;\n    // Box drawing style (optional)\n    style: Style | null;\n}\n\ninterface Style {\n    type: "Style";\n    corners: "rounded" | "sharp";\n}\n\n```\n'} {'role': 'user', 'content': "\nThe following is a user request:\n'''\ndraw two squares of side 50 in a diagonal\n'''\n\nThe following is the user request translated into a JSON object with 2 spaces of indentation and no properties with the value undefined:\n"} {   "items": [     {       "type": "Box",       "x": 0,       "y": 0,       "width": 50,       "height": 50,       "text": "",       "style": null     },     {       "type": "Box",       "x": 50,       "y": 50,       "width": 50,       "height": 50,       "text": "",       "style": null     }   ] } ~> make it three squares --------- NEXT REQUEST --------- {'role': 'user', 'content': '\nYou are a service that translates user requests into JSON objects of type "Drawing" according to the following TypeScript definitions:\n```\n// A drawing is a list of boxes. (We\'ll add other elements later, like arrows.)\ninterface Drawing {\n    items: Array;\n}\n\n// Use this type for input that match nothing else\ninterface UnknownText {\n    type: "Unknown";\n    // The text that wasn\'t understood\n    text: string;\n}\n\n// A rectangular box.\n// \n// The coordinate system has origin top left, x points right, y points down.\n// Measurements are in pixels.\n// \n// There can also be text in the box. There are optional style properties.\ninterface Box {\n    type: "Box";\n    // Top left corner coordinates\n    x: number;\n    y: number;\n    // Size of the box\n    width: number;\n    height: number;\n    // Text centered in the box\n    text: string;\n    // Box drawing style (optional)\n    style: Style | null;\n}\n\ninterface Style {\n    type: "Style";\n    corners: "rounded" | "sharp";\n}\n\n```\n'} {'role': 'user', 'content': "\nThe following is a user request:\n'''\ndraw two squares of side 50 in a diagonal\n'''\n"} {'role': 'assistant', 'content': '\nThe following is the user request translated into a JSON object with 2 spaces of indentation and no properties with the value undefined:\n```\n{\n  "items": [\n    {\n      "type": "Box",\n      "x": 0,\n      "y": 0,\n      "width": 50,\n      "height": 50,\n      "text": "",\n      "style": null\n    },\n    {\n      "type": "Box",\n      "x": 50,\n      "y": 50,\n      "width": 50,\n      "height": 50,\n      "text": "",\n      "style": null\n    }\n  ]\n}\n```\n'} {'role': 'user', 'content': "\nThe following is a user request:\n'''\nmake it three squares\n'''\n\nThe following is the user request translated into a JSON object with 2 spaces of indentation and no properties with the value undefined:\n"} retrying (ReadTimeout('')) ... retrying (ReadTimeout('')) ... {   "items": [     {       "type": "Box",       "x": 0,       "y": 0,       "width": 50,       "height": 50,       "text": "",       "style": null     },     {       "type": "Box",       "x": 50,       "y": 50,       "width": 50,       "height": 50,       "text": "",       "style": null     },     {       "type": "Box",       "x": 100,       "y": 100,       "width": 50,       "height": 50,       "text": "",       "style": null     }   ] } ~> label them according to size --------- NEXT REQUEST --------- {'role': 'user', 'content': '\nYou are a service that translates user requests into JSON objects of type "Drawing" according to the following TypeScript definitions:\n```\n// A drawing is a list of boxes. (We\'ll add other elements later, like arrows.)\ninterface Drawing {\n    items: Array;\n}\n\n// Use this type for input that match nothing else\ninterface UnknownText {\n    type: "Unknown";\n    // The text that wasn\'t understood\n    text: string;\n}\n\n// A rectangular box.\n// \n// The coordinate system has origin top left, x points right, y points down.\n// Measurements are in pixels.\n// \n// There can also be text in the box. There are optional style properties.\ninterface Box {\n    type: "Box";\n    // Top left corner coordinates\n    x: number;\n    y: number;\n    // Size of the box\n    width: number;\n    height: number;\n    // Text centered in the box\n    text: string;\n    // Box drawing style (optional)\n    style: Style | null;\n}\n\ninterface Style {\n    type: "Style";\n    corners: "rounded" | "sharp";\n}\n\n```\n'} {'role': 'user', 'content': "\nThe following is a user request:\n'''\ndraw two squares of side 50 in a diagonal\n'''\n"} {'role': 'assistant', 'content': '\nThe following is the user request translated into a JSON object with 2 spaces of indentation and no properties with the value undefined:\n```\n{\n  "items": [\n    {\n      "type": "Box",\n      "x": 0,\n      "y": 0,\n      "width": 50,\n      "height": 50,\n      "text": "",\n      "style": null\n    },\n    {\n      "type": "Box",\n      "x": 50,\n      "y": 50,\n      "width": 50,\n      "height": 50,\n      "text": "",\n      "style": null\n    }\n  ]\n}\n```\n'} {'role': 'user', 'content': "\nThe following is a user request:\n'''\nmake it three squares\n'''\n"} {'role': 'assistant', 'content': '\nThe following is the user request translated into a JSON object with 2 spaces of indentation and no properties with the value undefined:\n```\n{\n  "items": [\n    {\n      "type": "Box",\n      "x": 0,\n      "y": 0,\n      "width": 50,\n      "height": 50,\n      "text": "",\n      "style": null\n    },\n    {\n      "type": "Box",\n      "x": 50,\n      "y": 50,\n      "width": 50,\n      "height": 50,\n      "text": "",\n      "style": null\n    },\n    {\n      "type": "Box",\n      "x": 100,\n      "y": 100,\n      "width": 50,\n      "height": 50,\n      "text": "",\n      "style": null\n    }\n  ]\n}\n```\n'} {'role': 'user', 'content': "\nThe following is a user request:\n'''\nlabel them according to size\n'''\n\nThe following is the user request translated into a JSON object with 2 spaces of indentation and no properties with the value undefined:\n"} {   "items": [     {       "type": "Box",       "x": 0,       "y": 0,       "width": 50,       "height": 50,       "text": "Small",       "style": null     },     {       "type": "Box",       "x": 50,       "y": 50,       "width": 50,       "height": 50,       "text": "Medium",       "style": null     },     {       "type": "Box",       "x": 100,       "y": 100,       "width": 50,       "height": 50,       "text": "Large",       "style": null     }   ] } ~> oops, label them according to position --------- NEXT REQUEST --------- {'role': 'user', 'content': '\nYou are a service that translates user requests into JSON objects of type "Drawing" according to the following TypeScript definitions:\n```\n// A drawing is a list of boxes. (We\'ll add other elements later, like arrows.)\ninterface Drawing {\n    items: Array;\n}\n\n// Use this type for input that match nothing else\ninterface UnknownText {\n    type: "Unknown";\n    // The text that wasn\'t understood\n    text: string;\n}\n\n// A rectangular box.\n// \n// The coordinate system has origin top left, x points right, y points down.\n// Measurements are in pixels.\n// \n// There can also be text in the box. There are optional style properties.\ninterface Box {\n    type: "Box";\n    // Top left corner coordinates\n    x: number;\n    y: number;\n    // Size of the box\n    width: number;\n    height: number;\n    // Text centered in the box\n    text: string;\n    // Box drawing style (optional)\n    style: Style | null;\n}\n\ninterface Style {\n    type: "Style";\n    corners: "rounded" | "sharp";\n}\n\n```\n'} {'role': 'user', 'content': "\nThe following is a user request:\n'''\ndraw two squares of side 50 in a diagonal\n'''\n"} {'role': 'assistant', 'content': '\nThe following is the user request translated into a JSON object with 2 spaces of indentation and no properties with the value undefined:\n```\n{\n  "items": [\n    {\n      "type": "Box",\n      "x": 0,\n      "y": 0,\n      "width": 50,\n      "height": 50,\n      "text": "",\n      "style": null\n    },\n    {\n      "type": "Box",\n      "x": 50,\n      "y": 50,\n      "width": 50,\n      "height": 50,\n      "text": "",\n      "style": null\n    }\n  ]\n}\n```\n'} {'role': 'user', 'content': "\nThe following is a user request:\n'''\nmake it three squares\n'''\n"} {'role': 'assistant', 'content': '\nThe following is the user request translated into a JSON object with 2 spaces of indentation and no properties with the value undefined:\n```\n{\n  "items": [\n    {\n      "type": "Box",\n      "x": 0,\n      "y": 0,\n      "width": 50,\n      "height": 50,\n      "text": "",\n      "style": null\n    },\n    {\n      "type": "Box",\n      "x": 50,\n      "y": 50,\n      "width": 50,\n      "height": 50,\n      "text": "",\n      "style": null\n    },\n    {\n      "type": "Box",\n      "x": 100,\n      "y": 100,\n      "width": 50,\n      "height": 50,\n      "text": "",\n      "style": null\n    }\n  ]\n}\n```\n'} {'role': 'user', 'content': "\nThe following is a user request:\n'''\nlabel them according to size\n'''\n"} {'role': 'assistant', 'content': '\nThe following is the user request translated into a JSON object with 2 spaces of indentation and no properties with the value undefined:\n```\n{\n  "items": [\n    {\n      "type": "Box",\n      "x": 0,\n      "y": 0,\n      "width": 50,\n      "height": 50,\n      "text": "Small",\n      "style": null\n    },\n    {\n      "type": "Box",\n      "x": 50,\n      "y": 50,\n      "width": 50,\n      "height": 50,\n      "text": "Medium",\n      "style": null\n    },\n    {\n      "type": "Box",\n      "x": 100,\n      "y": 100,\n      "width": 50,\n      "height": 50,\n      "text": "Large",\n      "style": null\n    }\n  ]\n}\n```\n'} {'role': 'user', 'content': "\nThe following is a user request:\n'''\noops, label them according to position\n'''\n\nThe following is the user request translated into a JSON object with 2 spaces of indentation and no properties with the value undefined:\n"} retrying (ReadTimeout('')) ... {   "items": [     {       "type": "Box",       "x": 0,       "y": 0,       "width": 50,       "height": 50,       "text": "First",       "style": null     },     {       "type": "Box",       "x": 50,       "y": 50,       "width": 50,       "height": 50,       "text": "Second",       "style": null     },     {       "type": "Box",       "x": 100,       "y": 100,       "width": 50,       "height": 50,       "text": "Third",       "style": null     }   ] } ~> ```
DanielRosenwasser commented 2 months ago

There's some stuff here that makes intuitive sense, but I'll need to check in with others (CC @ahejlsberg)

I would think that the best format for incorporating chat history would be something like

That gives background for the current convo, plus reinforces the current task at hand. This PR is pretty close to that.

gvanrossum commented 2 months ago

I wonder how to proceed. Clearly this PR two things (adding history and adding a new example) and that's not great. Your fixes to the translator (#240) broke my PR and I would rather not try to merge it back. I propose to add my example without chat history and start a separate discussion on chat history -- okay?