-
### Bug description
I would like to use DeviceStatsMonitor during a trainer.test() call. I followed the [relative documentation](https://pytorch-lightning.readthedocs.io/en/stable/tuning/profiler_b…
-
E.g., for https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/periodic_elements there is no score plotted, but the score is present in human evaluation file.
Randl updated
2 years ago
-
### Project URL
GitHub: https://github.com/google/BIG-bench
PyPI: https://pypi.org/project/bigbench
### Does this project already exist?
- [x] Yes
### New limit
0.4GB
### Update iss…
-
In the result plots at the bottom of the README of [logical_deduction](https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/logical_deduction), the random baselines are off: for all …
-
In the [instance-level logs](https://github.com/google/BIG-bench/blob/main/bigbench/benchmark_tasks/arithmetic/results/full-logs_BIG-G_BIG-G-sparse.tar.lrz) for the [arithmetic](https://github.com/goo…
-
I'm getting a weird error named when executing
```
pablo$ scripts/evaluate_task.sh --task multistep_arithmetic --output_path results.json
Traceback (most recent call last):
File "bigbench/evalua…
-
I'm not sure whether we need to review the assumption of "[Training on the test set](https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/training_on_test_set)" task or there is inde…
Randl updated
2 years ago
-
See, e.g., https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/chess_state_tracking
One possible solution is to put legend outside of plot.
Randl updated
2 years ago
-
I'm following the instructions on this webpage:
https://github.com/google/BIG-bench/blob/main/README.md#how-do-i-create-a-task
There's a section that says:
> **Testing and evaluating**
>
> Onc…
-
See https://github.com/google/BIG-bench/blob/main/bigbench/benchmark_tasks/simple_arithmetic/results/scores_GPT_GPT-3-200B.json#L33
Randl updated
2 years ago