feedback

[mrT23]

Hi @DanielRosenwasser

You don't have a 'discussion' tab, so I am abusing a bit the 'issue' tab :-)

I am Tal from Codium. Following our talk previously today at the expo, I experimented a bit with your repo and methodology. sharing my results and feedback:

https://docs.google.com/document/d/1_MUKWbugyHN1p4o3xAu3HwsGv2VacJ5bTls2ipL837s/edit?usp=sharing

[mrT23]

p.s.

after the talk today in the conference, i wonder if what your are suggesting is not already implemented by openai function calls https://www.datacamp.com/tutorial/open-ai-function-calling-tutorial (and https://github.com/jxnl/instructor)

and also maybe if pydantic is not a more suitable way to represent the input

[DanielRosenwasser]

Thanks again for the feedback @mrT23. I'd say when it comes to OpenAI function calling:

1. It's tied to OpenAI - which is fine for a lot of people, but you may need other providers or options
2. Functions are just fine-tuned. You still need to validate that you got proper JSON, and you need something like Instructor or Pypechat/TypeChat to make sure that the shapes/types actually line up.

3. Function calling seems to be targeted more at performing tasks, rather than more general-purpose translation to whatever objects you want.

   When it comes to performing tasks (also called scripting and planning), we have TypeChat programs. Open AI functions operate at a single function at a time, whereas TypeChat programs are single-shot to construct, but can perform multiple steps.

Instructor is definitely another very cool library taking a similar approach. One difference is that its spec language is JSON schema, and Pypechat currently just uses Python syntax. We might experiment further here.

Another difference is that we have TypeChat programs which are API-based and provide validation across steps. You can see examples of that in the math demo and the dataframes demo.

[mrT23]

@DanielRosenwasser I mostly agree with you. i am not a big fan of using JSON scheme - they are hard to read and write, take a lot of tokens, and I am not sure the model excels at deciphering them.

Defining an input with a simple python structure is very easy to read and maintain, economical in terms of tokens, and models (openAI and others) handle it well.

It also enables you to use YAML and not JSON, which is more suitable for code-like outputs.

I do think you should consider using Pydantic - its the most common python library for this kind of tasks.

Following the conference and our dialog, i am switching most of my prompts to this nice structure:

The output must be a YAML object equivalent to type $Behaviours, according to the following Pydantic definitions:
'
class Behaviour(BaseModel):
    base_behaviour: str
    sub_behaviours: list[str] = Field(min_items=2, max_items=4, description=...")

class Behaviours(BaseModel):
    behaviours: list[Behaviour] = Field(min_items=..., max_items=...)
'

Example YAML output:
\```yaml
behaviours:
- base_behaviour: |
      ...
  sub_behaviours:
  - |
    sub behaviour ...
  - |
    sub behaviour ...
...

The first part defines the structure, and the second part further helps the model (like a one-shot example), but also makes sure to maintain some YAML peculiarities, like block scalar.

Using JSON instead of YAML just means changing the 2nd part