Open KlaudiaTH opened 1 year ago
Any perplexity tasks fail, same goes for the German ones. The issue is that the EvalHarnessAdaptor
in Megatron-DeepSpeed/tasks/eval_harness/evaluate.py
does not implement the greedy_until
method, which perplexity tasks require.
Happened also for germanquad and squad2
New images ... Taurus: /projects/p025/p_gptx/apptainer_images/obmd-lmeval-21.12_100423-py3.sif Juwels: /p/scratch/opengptx-elm/shared/apptainer_images/obmd-lmeval-21.12_100423-py3.sif
@katrinklug
When running the tasks mlqa_en,mlsum_en, xquad_en and xquad_en I get the following error message:
Performed evaluation on Taurus using scripts from evaluation repository: apptainer/juwels_german-evalds.sbatch