Experimentation mode

[shreyashankar]

Maybe have a context manager?

• Need a way to inject params & automatically hash the params to create instance names (to leverage the cache)
• Need a way to expose the instance names
• Integrations with experiment trackers
• Evaluator ops
• Tunable prompts
• Ability to easily clear the cache

[shreyashankar]

See https://github.com/shreyashankar/prompteng/blob/main/travelagent.py for a very clunky example of user having to hash the experiment variables.

[shreyashankar]

Possibly, each component state comes with an OpenAI or LLM connection, and we have nice utils to query the LLM (with retry decorators, role the LLM must play, parsing functions, etc). Or we have our own LLMComponent object that users can put in their component states, which encompass roles, parsing functions, etc. LLMComponent objects should not have any data!

LLM role:

• You are a [x]

Parsing functions:

• Yes or no
• JSON
• Summary

[shreyashankar]

Experimentation

Goal: for a component, a user should be able to define a dataset, initialize a template in init_state, iteratively refine a template in a jupyter notebook, and then launch an experiment to find the best prompt template for their dataset. When the component is running in prod, the experiment should be able to run automatically, under the hood.

Iterative Prompt Exploration

• User creates a component & puts a dataset in the state.
• User puts an initial template in the state.
• User creates a flow A with infer op to leverage the template on the examples & fit op to evaluate the results of all the infer ops (so we need to allow a batch size of none). Fit op updates the template in the state.
• User creates an experiment context, where they run the flow for each example. The experiment context tracks the history of the state for that session.
• User can then iteratively refine the prompt.

Auto Prompt Exploration

(Start from a template found by the process above)

• Write a flow B to run an entire experiment. In the infer op, we create a connection to itself & run flow A on the dataset with some hparams. In the fit op, we save the result of the experiment to the state.
• User runs flow B with all the ideas they want to try. Flow B runs in parallel for all the ideas, within a process pool. The experiment results are added to the state, inspected on finish.

Things to add

• motion dataset class: wrapper around a dataframe with train, dev, test accessors
• allow batch size of None
• figure out how to pass instance id & params at the component level (?)
• iterativeML experiment context to track history of the state (not logged anywhere)
• autoML experiment context to run many ideas in parallel, using a process pool

[shreyashankar]

Consider always setting batch size to 1? Might be easier to manage tbh