Add a config file for experiments

[recitalAI]

Too many parameters are shared between the different generation steps. It would be more convenient to use a configuration file to define all the steps and run them at once.

[GillesMoyse]

In progress (cf. PR from Youssouf)

[GillesMoyse]

The configuration file describes how an Expe will go from one step to another, in which order. Thus, the configuration file is made of the 3 main blocks :

1. an Expe file
2. a dictionary of Steps
3. a workflow, an ordered list of Steps

Expe file : a string to the Expe file

Each Step is defined as follows:

• 1 TextGenerator
• export parameters (json, html, spreadsheet)

Each TextGenerator comprises:

• a Prompter
• one or several LLMs - LLMs can be defined as classes or names - if defined as names, they are instances of LiteLLM
• a set of parameters (overwrite, start_from, missing_only)

Below is an example of a configuration file with 3 steps, AnswerGen, FactsGen and EvalGen:

• expe_file
• expe_folder
• steps: {
  • "AnswerGen": {
  • generator: "AnsGenerator" # TextGenerator class
  • params: {"overwrite", "start_from", "b_missing"}
  • prompter: "PptrAnsBase"
  • retriever: None
  • llm_names: ["gpt-4o", "mistral-large"]
  • llms: ["Albert"]
  • export: {
    • "json": {"do_export": true}
    • "htrml": {"do_export": true, "template": "basic.jinja"}
    • "spreadsheet": {"do_export": false} }
  • "Facts
  • ... }
• workflow: ["AnswerGen", "AsnwerGen", "Facts"]

pipeline = load_pipeline(path_to_pipeline.toml) pipeline.run()