While trying to run the steps given in leaderboard README, found following improvements
1-Setup
model variable to be initialised before creating generations and metrics directories
2-Generations
save_generations flag is missing in while running generationsmax_length to be 1024 for some tasks, based on your tokeniser (Fix for #207)
While trying to run the steps given in leaderboard README, found following improvements
1-Setup
modelvariable to be initialised before creating generations and metrics directories2-Generations
save_generationsflag is missing in while running generationsmax_lengthto be 1024 for some tasks, based on your tokeniser (Fix for #207)3-Evaluations Generations file is saved in
save_generations_path_$task, while running evaluations it should load from this path(_$task is missing in the path in README). https://github.com/bigcode-project/bigcode-evaluation-harness/blob/094c7cc197d13a53c19303865e2056f1c7488ac1/main.py#L387