Log results in raw form before doing any smart averaging so that the full results are always logged in case there are errors later. Use a task metric map to avoid having to use specific metric names in averaging code. Currently only supports tasks in the paper and MMLU. New tasks and the corresponding metric of interest can be added where necessary.
Log results in raw form before doing any smart averaging so that the full results are always logged in case there are errors later. Use a task metric map to avoid having to use specific metric names in averaging code. Currently only supports tasks in the paper and MMLU. New tasks and the corresponding metric of interest can be added where necessary.