Closed renebidart closed 3 years ago
It seems there are some discussions under https://github.com/tensorflow/tensorflow/issues/9489. Some have found that it could be a GPU memory issue, maybe good to try out if the issue still persists with a small version of the dataset.
Thanks, using tf.config.experimental.set_visible_devices([], "GPU") fixed it.
Hello, I'm running into a strange issue training models (performer, bigbird, longformer) on listops. The model works fine on CPU, but on GPU (either on one or multiple v100 16GB) it crashes with a strange error. This doesn't happen with any other dataset I've tried in the benchmark. The error is:
The dataset was created with the included script:
PYTHONPATH="$(pwd)":"$PYTHON_PATH" python lra_benchmarks/data/listops.py --output_dir=lra_data/listops
Any idea what may be causing this?