-
(gpt5) [jfreeman@mir-81 promptbase]$ python -m promptbase mmlu
usage: __main__.py [-h] [--subject SUBJECT] [--list_subjects] {gsm8k,humaneval,math,drop,bigbench}
__main__.py: error: argument dataset…
-
Hi would be cool to valuate all openai models on Beyond the Imitation Game Benchmark (BIG-bench) which is a collaborative benchmark intended to probe large language models and extrapolate their future…
-
Hi, I'm running BIG-Bench Lite tasks, succeeded in running 22 of 24 tasks, and for the two tasks language_identification and logic_grid_puzzle I get segmentation fault.
I am using the main branch of …
-
Hi @andersjohanandreassen and team :)
At Hugging Face we find Big-Bench super useful and we'd like to allow anyone to access, explore and use any task. An effort was already lead by the awesome @an…
-
Here is the output from my prompt_spec:
```
Instruction:
Task description: Identify a broad class given several examples from that class
Examples:
input=
Q: The similarity among DemandBase, In…
-
As noticed in https://github.com/huggingface/datasets/pull/4125 when a dataset config class has a parameter that reduces the number of examples (e.g. named `max_examples`), then loading the dataset an…
-
Hello, Are you still paying attention to this project?
I execute it in your way: ./bin/bigBench runBenchmark -d bigbench -f 2 -m 80 -s 6 -b -e spark_sql
But, the command line keeps showing Error: th…
-
Hi, there seems to be a discrepancy between the GPT models described in the paper and the ones reported in the results.
According to the paper:
> We use OpenAI GPT models corresponding to the G…
-
The tutorial example at https://microsoft.github.io/Trace/examples/nlp/bigbench_hard.html seem to contain two bugs (one being more of a typo because it does not affect the training):
1. The optimiz…
-
Firstly, check the `English proverbs` dataset:
https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/english_proverbs
![image](https://github.com/google/BIG-bench/assets/8592144/6df…