Closed quantitative-technologies closed 2 years ago
I see where the problem comes from. Will push a fix tonight or tomorrow morning, then we will do a patch release. In the meantime you should have no error by staying on v4.10
I run out of memory using transformers v4.X where X > 10 training led-large-16384-arxiv
with four gradient accumulation steps and a batch size of two like in this notebook on an A6000 with 48 GB of RAM. I had to bump gradient accumulation steps and batch size down to 1 each to fit the model + batch on the GPU. Wild. Don't really feel like opening an issue, but yeah just thought I'd chirp in here and say that with v4.10.1 I can fit up to 8 samples per batch with four gradient accumulation steps on the A6000.
If you upgrade to 4.11.1
in the colab notebook I shared it fails, but for 4.10.1
it works just fine.
Environment info
transformers
version: 4.12.0.dev0Who can help
@sgugger
Information
The example script below was working fine until today. I believe that it was working in version
4.11.0.dev0
. If you can please tell me how to checkout the source for4.11.0.dev0
from github, I will confirm that it works.To reproduce
Steps to reproduce the behavior:
On a TPU colab instance with High-RAM, run:
Gives the error:
Expected behavior
No error.