Closed yongzx closed 2 years ago
I've had it happen before, but just trying again later went much faster and finished quickly.
Which language was this? I am debugging it right now, and have been able to replicate this bug consistently.
This was with Thai. I’ll try with Burmese
I ran this with Burmese, extending vocab, 100k oscar samples, and 24k vocab size.
It hung for a while after finishing counting pairs though. Also I think checked and I think that this script was using 100gb+ (VIRT in top
) so maybe your bug is that you run out of CPU memory or RAM.
Thanks Hailey! Yeap it's memory issue.
Have you been able to replicate this? I've had it happen before, but just trying again later went much faster and finished quickly.