Should FunctionCall be part of the Expression?

[caoer]

The following schema is introduced in https://github.com/microsoft/TypeChat/pull/20.

// A program consists of a sequence of expressions that are evaluated in order.
export type Program = {
    "@steps": Expression[];
}

// An expression is a JSON value, a function call, or a reference to the result of a preceding expression.
export type Expression = JsonValue | FunctionCall | ResultReference;

// A JSON value is a string, a number, a boolean, null, an object, or an array. Function calls and result
// references can be nested in objects and arrays.
export type JsonValue = string | number | boolean | null | { [x: string]: Expression } | Expression[];

// A function call specifices a function name and a list of argument expressions. Arguments may contain
// nested function calls and result references.
export type FunctionCall = {
    // Name of the function
    "@func": string;
    // Arguments for the function
    "@args": Expression[];
};

// A result reference represents the value of an expression from a preceding step.
export type ResultReference = {
    // Index of the previous expression in the "@steps" array
    "@ref": number;
};

However, I don't think FunctionCall should be part of Expression type. Because { [x: string]: Expression } is part of JsonValue. The output of gpt model is "@steps": Expression[], the execution order is determined. Because when we allow FunctionCall in an object-like value, the execution order is undetermined. Example:

with FunctionCall

{
  "@steps": [
    {
      "@func": "func1",
      "@args": [
        {
          "a": {
            "@func": "func2",
            "args": []
          },
          "b": {
            "@func": "func3",
            "args": []
          }
        }
      ]
    }
  ]
}

we don't know the execution order of func2 vs func3

with Reference

{
  "@steps": [
    {
      "@func": "func3",
      "args": []
    },
    {
      "@func": "func2",
      "args": []
    },
    {
      "@func": "func1",
      "@args": [
        {
          "a": {
            "@ref": 0
          },
          "b": {
            "@ref": 1
          }
        }
      ]
    }
  ]
}

func3 is executed first, then func2.

---

I'm not sure in what scenarios the gpt model can produce @steps with reference to FunctionCall, in the tests I run, only ResultReference is produced. Is gpt model able to understand the difference between reference to the result of previous functioncall and execute every times it is evaluated?

I think it would be 'safer' to design the generic schema as deterministic as possible. For example, in a financial application, the execution orders matters for the state mutation for user's funds.

therefore, a more strict schema version is:

export type Program = {
    "@steps": FunctionCall[];
}

export type Expression = JsonValue | ResultReference;

export type JsonValue = string | number | boolean | null | { [x: string]: Expression } | Expression[];

export type FunctionCall = {
    "@func": string;
    "@args": Expression[];
};

export type ResultReference = {
    "@ref": number;
};

@ahejlsberg @steveluc does it make sense? Have you seen any example with including FunctionCall in Expression?

[ahejlsberg]

In principle I agree, anything you can do with nested function calls you can do through top-level function calls in discrete steps. However, it is hard to convince gpt-3.5-turbo to do that in all cases. For example, evaluating the expression 1+2*3 using the math example causes gpt-3.5-turbo to produce the following JSON program:

{
  "@steps": [
    {
      "@func": "add",
      "@args": [
        1,
        {
          "@func": "mul",
          "@args": [
            2,
            3
          ]
        }
      ]
    }
  ]
}

In our experience, instructing gpt-3.5-turbo to only generate top-level function calls doesn't work well. We've observed the model getting confused and generating incorrect programs. That said, such a restriction doesn't seem to be a problem for gpt-4.

[ahejlsberg]

BTW, execution order is definitely deterministic either way. It's just not as obvious with the nested function calls.

[caoer]

It is indeed the case for gpt-3.5-turbo, I was testing mostly on gpt-4 so I was noticing it yet. Thanks very much for sharing the info.

Regarding the the deterministic of execution order, I still do believe it is not, if we consider this: { [x: string]: Expression }.

Take following as example:

{
  "@steps": [
    {
      "@func": "func1",
      "@args": [
        {
          "a": {
            "@func": "func2",
            "args": []
          },
          "b": {
            "@func": "func3",
            "args": []
          }
        }
      ]
    }
  ]
}

func1 take 1 parameter of object: {a: ..., b: ...}. a and b is the result of another function. Since it is inside an object. we don't know if execution order is func2 -> func3 -> func1 or func3 -> func2 -> func1

---

overall, the more strict version with function call in the steps can be set as following. not saying it is more suitable in the context of interacting with gpt model. just want to throw it onto the table for discussion.

---

export type Program = {
  "@steps": FunctionCall[];
};

export type Expression = JsonValue | ResultReference;

export type JsonValue =
  | string
  | number
  | boolean
  | null
  | { [x: string]: Expression }
  | (Expression | FunctionCall)[];

export type FunctionCall = {
  "@func": string;
  "@args": (Expression | FunctionCall)[];
};

export type ResultReference = {
  "@ref": number;
};

I'm not sure if it is GPT friendly. Especially with the comments that are tailor to tech the GPT the concept of 'Expression'.