search

microsoft / task_oriented_dialogue_as_dataflow_synthesis

Code to reproduce experiments in the paper "Task-Oriented Dialogue as Dataflow Synthesis" (TACL 2020).
MIT License
308 stars 66 forks source link

How could one execute a dataflow program? #62

Closed kingb12 closed 2 years ago

kingb12 commented 2 years ago

Hi all! Apologies in advance for the wall of text. I was interested in looking into interactive extensions using the dataflow representation and wanted to know roughly how executor of a lispress program works in a fully interactive dialogue. To my understanding an executor is not provided, but I'm curious about how one could be constructed. The high-level dialogue flow I considered was:

  1. Parse a user utterance and context history into a new Program
    • all the important pieces for this seem to be provided
  2. From this Program, use infer_types(program: Program, library: Dict[str, Definition]) -> Program to produce an equivalent Program that is fully-typed
    • Missing for SMCalFlow is the library: Dict[str, Definition] which if I understand right, cannot be provided due to legal reasons mentioned here.
  3. Execute this fully-typed lispress Program to produce an execution ?
  4. Generate an agent utterance from the program and its execution
  5. Use these results as context in subsequent dialogues

Most of my questions are on representations for 3 and 5:

Thanks for releasing this dataset and library and for any insight you can provide! For context, given the size of the SMCalFlow library that would need to be implemented I am looking into other potential libraries to work with in the dataflow representation, but wanted to know the best place to start.

sammthomson commented 2 years ago
  • Is this a reasonable understanding of the process for a full interactive system in this representation?

Yep.

Steps 1 and 2 could be combined, and you could predict fully typed Lispress directly if you wanted.

The executor also has access to the the dataflow graph (which encodes the history of program executions), and it returns an extended dataflow graph.

Any computations done while computing the agent utterance and their results are also appended to the dataflow graph.

  • For type-inference and execution of the program in step 2 & 3, I believe one would need a library: Dict[str, Definition] with additional pointers to implementations and some kind of executor execute(program: Program, library: Dict[str, Definition]) -> ? which could be library-agnostic.
    • Is this correct, or are there additional missing pieces not present in this repository?

That's right, just missing the dataflow graph. Something like:

execute(program: Program, library: Dict[str, Definition], dataflow: DataflowGraph) -> DataflowGraph
  • What type of result would be produced by an executor? Could one just append the result of each CallOp as a ValueOp expression, returning an extended Program?

Each dataflow graph node has both the CallOp info that produced it and the value that it resulted in. The dataflow graph also includes all the subroutine computations and results, sort of like if all the function calls in the program had been inlined.

  • Is a single Program also an adequate representation for a full dialogue, or only a turn? e.g. each turn in a dialogue appears to produce a distinct Program instead of appending expressions to the initial one, but I wasn't sure if this was a dataset artifact or something that would also be true when run interactively.

A Program is meant to represent only the turn it corresponds to. The dataflow graph (not publicly released) is the append-only representation of the full dialogue.

  • Last, the execution methods in the multiwoz section of the repository seem to be focused on translating a Program to a belief state dictionary and vice-versa. Do these also resemble actual program execution in dataflow? For example the VanillaSalienceModel takes an ExecutionTrace argument which itself holds slot-value pairs. What would a similar VanillaSalienceModel look like in a dataflow-only representation?

A salience model in dataflow takes a constraint that represents what the user is referring to as an explicit argument, the dataflow graph as an implicit argument, and returns a dataflow node. It can be a learned model. It could also consult external sources, e.g. if a name "Sandy" is mentioned, but has not been previously mentioned in the dialog, it could call a people-search api, appending to dataflow in the process, and return the newly created node.

Thanks for releasing this dataset and library and for any insight you can provide! For context, given the size of the SMCalFlow library that would need to be implemented I am looking into other potential libraries to work with in the dataflow representation, but wanted to know the best place to start.

AFAIK we are the only ones doing things this way, so you might not find any other existing datasets that match this interface and also include library definitions.

kingb12 commented 2 years ago

Thanks for the response, this was extremely helpful!