-
This is pertaining to the tasks listed in - https://github.com/google/BIG-bench/blob/main/bigbench/benchmark_tasks/logical_deduction/three_objects/task.json
For example, the first task here is -
…
-
### Feature Type
- [ ] Adding new functionality to pandas
- [X] Changing existing functionality in pandas
- [ ] Removing existing functionality in pandas
### Problem Description
I wis…
-
The tutorial example at https://microsoft.github.io/Trace/examples/nlp/bigbench_hard.html seem to contain two bugs (one being more of a typo because it does not affect the training):
1. The optimiz…
-
I just use the colab demo for "Load and inspect a BIG-bench SeqIO Task" to inspect some cases from "arithmetic/4_digit_division"
```python
task = seqio.get_mixture_or_task(
"bigbench:arithme…
-
Hi,
Thank you for open-sourcing your code.
I was trying to run your code using Mistral and initially, there was an error that
```
ValueError: Asking to pad but the tokenizer does not have a p…
-
Thanks for the great work! I'm also using FLAN for training, so I'm wondering how to include only tasks that are in Tasksource but not in FLAN.
-
For the Legal Move task, I can't reproduce experimental results. I believe that the issue might be related to the order of training, so I would like to ask about the training details of the ChessGPT m…
-
### What
I am interested in training a `PHATGOOSE` model with several experts with datasets hosted in Hugging Face. How can I do that?
-
I am currently verifying all tasks under the `lm-evaluation-harness`. I will raise the issues I encounter one after another in this issue thread. Thank you for your inspection and response! @haileysch…
-
Hey, great to see a table of results like this, but one thing I noticed was that it seems like temperatures aren't specified on the table. Seems like it might be a good thing to specify since from the…