Closed Sumsky21 closed 2 years ago
Addition: Previously I've used a similar script to train BertForMaskedLM, and no memory fluctuations were observed that time:
That's why I suspect it's abnormal this time.
So the issue only appears with Electra, right? Did you encounter it for any other model?
Yes. I've only tried on Bert and Electra so far (and there's no problem during Bert training, as above figure has shown).
If there's a need, I can try the similar script on RobertaforMaskedLM in next few days.
That would be helpful if you could!
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.
Please note that issues that do not follow the contributing guidelines are likely to be ignored.
Environment info
transformers
version: 4.17.0Who can help
Electra model: @LysandreJik
Trainer: @sgugger
Information
Model I am using (Bert, XLNet ...): Electra(ForMaskedLM)
The problem arises when using:
The tasks I am working on is:
Expected behavior
I don't know if it was a bug, so I'll describe the situation first.
I'm using the trainer API to train a ElectraForMaskedLM model from scratch. I write a simple script (will be shown below) and set
save_step
argument as 10000. But after running the script, I observe the RAM(memory, NOT CUDA) usage continuously increases, finally casued an OOM error thus terminate the docker. At the same time, GPU, CUDA memory and CPU usage is quite normal.To solve the problem, I've tried adjusting several training arguments, and then find that when I set a smaller
save_step
(like 2000), RAM usage goes up and down periodically, and the cycle length is approximately equal to the interval between two save checkpoints, like the figure below:So is there a bug about the checkpoint saving mechanism? Or it's normal so I should have set a small
save_steps
?The training script is below:
To reproduce
Steps to reproduce the behavior:
save_steps
argument