Extract workflow trees using Chain of Code

[abrichr]

Feature request

We would like to convert a recording of one or more workflow iterations into a repeatable and versionable representation, ie code.

From https://chain-of-code.github.io/paper.pdf :

Code provides a general syntactic structure to build complex programs and perform precise computa- tions when paired with a code interpreter – we hypothesize that language models (LMs) can leverage code-writing to improve Chain of Thought reasoning not only for logic and arithmetic tasks [1, 5, 26], but also for semantic ones (and in particular, those that are a mix of both). For example, consider prompting an LM to write code that counts the number of times it detects sarcasm in an essay: the LM may struggle to write an implementation for “detect_sarcasm(string)” that can be executed by the interpreter (han- dling the edge cases would be insurmountable). However, LMs may still produce a valid solution if they not only write code, but also selectively “emulate” the interpreter by generating the expected output of “detect_sarcasm(string)” and other lines of code that cannot be executed. In this work, we propose Chain of Code (CoC), a simple yet surprisingly effective extension that improves LM code-driven reason- ing. The key idea is to encourage LMs to format semantic sub-tasks in a program as flexible pseudocode that the interpreter can explicitly catch undefined behaviors and hand off to simulate with an LM (as an “LMulator"). Experiments demonstrate that Chain of Code outperforms Chain of Thought and other baselines across a variety of benchmarks; on BIG-Bench Hard, Chain of Code achieves 84%, a gain of 12% over Chain of Thought. CoC scales well with large and small models alike, and broadens the scope of reasoning questions that LMs can correctly answer by “thinking in code".

Motivation

Versionability, reapeatability, stability, and verifiability of workflow definitions.

Chain of Code has several attractive properties:

1. It enables code use in entirely new regimes, by combining the advantages of code with the powerful semantic and commonsense knowledge of language models, which can easily express rules that are challenging to express in code (e.g., which foods are fruits?). Such an ability may have benefits beyond reasoning problems and its flexibility enables executing expressive language, such as pseudocode.
2. It leverages the ability of language models to code, a particular strength of recent language models due to the high quality data available.
3. It inherits many of the benefits of reasoning code, both the formal yet expressive structure of code (e.g., Turing completeness) and powerful computational tools available to code (whether simply multiplying two numbers, calculating √5 12121, or simulating physics).
4. It inherits many of the benefits of techniques that reason via intermediate steps, such as Chain of Thought. These techniques enable the language model to use more computation when necessary to solve a problem as well as provide more interpretability.

[abrichr]

Possibly related: https://data-science-blog.com/blog/2023/11/15/object-centric-process-mining-on-data-mesh-architectures/

[abrichr]

Closing in favor of https://github.com/OpenAdaptAI/OpenAdapt/issues/564