search

microsoft / promptbase

All things prompt engineering
MIT License
5.35k stars 293 forks source link

Missing mmlu as valid_dataset even though readme encourages download #21

Open jackfreeman88 opened 8 months ago

jackfreeman88 commented 8 months ago

(gpt5) [jfreeman@mir-81 promptbase]$ python -m promptbase mmlu usage: main.py [-h] [--subject SUBJECT] [--list_subjects] {gsm8k,humaneval,math,drop,bigbench} main.py: error: argument dataset: invalid choice: 'mmlu' (choose from 'gsm8k', 'humaneval', 'math', 'drop', 'bigbench')

Head of main.py :

import argparse from promptbase.gsm8k import gsm8k from promptbase.humaneval import humaneval from promptbase.math import math from promptbase.drop import drop from promptbase.bigbench import bigbench from promptbase.bigbench.consts import BIGBENCH_SUBJECTS

VALID_DATASETS = ["gsm8k", "humaneval", "math", "drop", "bigbench"]

:/

riedgar-ms commented 8 months ago

I'm looking in to this as part of #23