Eval framework: Write summary.json for easy access of metrics by programs.
Example:
r = pg.load('/path/to/summary.json', force_dict=True)
for task_name, task_results in r.items():
print(task_name):
for tr in task_results:
print('PROMPT', tr.experiment.prompt.template_str)
if tr.metrics is not None: # Finished experiment.
print('ACCURACY', tr.metrics.match_rate)
NOTE: Users could either do pg.load('/path/to/summary.json) to load the results with original experiment definition (require importing dependent code), or use pg.load('/path/to/summary.json', force_dict=True) to load the experiment definition and output as dicts, without import experiment definition.
Eval framework: Write
summary.jsonfor easy access of metrics by programs.Example:
NOTE: Users could either do
pg.load('/path/to/summary.json)to load the results with original experiment definition (require importing dependent code), or usepg.load('/path/to/summary.json', force_dict=True)to load the experiment definition and output as dicts, without import experiment definition.