exp: create a method [similar to eval()] to run and compare multiple prompts on single data file input. - Githubissues

AutoResearch / autodoc

MIT License

0 stars 1 forks source link

exp: create a method [similar to eval()] to run and compare multiple prompts on single data file input. #31

Closed anujsinha3 closed 7 months ago

anujsinha3 commented 7 months ago

eval() is a command-line typer method that takes 'prompt_id'. This prompt_id is used to select the inference prompt that should run on the data_file.

This task involves, creating a method similar to eval() that can:

take a list of prompts (or prompt_ids) as input
for each prompt - run eval_prompt() for data_file
for each prompt - log the results and parameters.
compare the performance for each run by using evaluation metrics