Closed fen0s closed 3 years ago
There are two variants of V100 - 16Gb and 32Gb. You need 32Gb. I can recommend https://immers.cloud service for that (I'm not affiliated).
I just don't really understand why GPT-2 Large is not runnable while original GPT-2 774M runs fine?.. Maybe I'm doing something wrong or there is a way to compress the model for 16 GB VRAM?
Runs or trains? You need way less memory to run a model.
Train, I meant. GPT-2 large trains fine and actually fast enough on V100 with gpt2-simple library, but the model of same size from sberbank isn't available for finetuning, weirdly enough. With same weights... What's the difference if architecture is basically identical and the only difference is training sets?
I've taken a quick look at gpt2-simple. It seems they use SGD and gradient checkpoints to fit the large model. I'd suggest you to set the batch_size to 1, optim to SGD and fp16 with O2 and see how it goes.
UPDATE: It did work! O2 arg for fpt16 finally fixed the not enough memory error. But now there's another issue that I have no idea how arises. Honestly, the whole dataset loading seems faulty, for some reason, considering it's second error with it.
@king-menin, mind helping to figure out what it is?
UPDATE 2: The features file generated seems to contain something really weird with seemingly broken encoding. Maybe that's the problem?
Update #3: The line_by_line allows you to start finetuning for a few iterations, but apparently, on some line it stops. Does it mean that some lines from the dataset just can't be loaded?
Update #4: Yeah, seems like it. Dataset with russian horror stories worked, while dataset of articles from foundation didn't. Seems like we'll need a function to preproccess dataset, apparently...
PROBLEM SOLVED: GPT-2 model needs formatting. You can't just feed it raw text. You need to use <|n|>
delimiter for your dataset, else it doesn't recognize it.. Honestly, you should update the pretraining section of readme, because this thing cost me 2 days of headache...
I don't quite understand the reason honestly. Colab provides V100 if you're a premium user, and I tried to run GPT-2 Large training (with fp16 and batch size 1), but it still runs out of memory. Original GPT-2 774M and even 1.5B were finetuning just perfectly. What's exactly different in russian model?