Open salykova opened 9 months ago
I'll take a look. Thanks for the report!
Is this fix in any way? I'm also encountering this bug, I've tried to add padding/truncate for my context length but the highest it will go is 8000 iter.
Transfering this to lightning since this file no longer exists, but there's still an underlying bug
milestone
May I ask have you address it? Or any quick fix for pretraining on opentext?
Hi all, when running
openwebtext_trainer.py
with default settings, the program crashes after 6k steps with following message:Openweb data was generated using
prepare_openwebtext.py
. Do you maybe know what causes the error?pretrain/openwebtext.py
however works fine.cc @carmocca