Why is it impossible to finetune GPT-2 Large on V100?..

ai-forever / ru-gpts

Russian GPT3 models.

Apache License 2.0

2.08k stars 444 forks source link

Why is it impossible to finetune GPT-2 Large on V100?.. #16

Closed fen0s closed 3 years ago

fen0s commented 3 years ago

I don't quite understand the reason honestly. Colab provides V100 if you're a premium user, and I tried to run GPT-2 Large training (with fp16 and batch size 1), but it still runs out of memory. Original GPT-2 774M and even 1.5B were finetuning just perfectly. What's exactly different in russian model?

mgrankin commented 3 years ago

There are two variants of V100 - 16Gb and 32Gb. You need 32Gb. I can recommend https://immers.cloud service for that (I'm not affiliated).

fen0s commented 3 years ago

I just don't really understand why GPT-2 Large is not runnable while original GPT-2 774M runs fine?.. Maybe I'm doing something wrong or there is a way to compress the model for 16 GB VRAM?

mgrankin commented 3 years ago

Runs or trains? You need way less memory to run a model.

fen0s commented 3 years ago

Train, I meant. GPT-2 large trains fine and actually fast enough on V100 with gpt2-simple library, but the model of same size from sberbank isn't available for finetuning, weirdly enough. With same weights... What's the difference if architecture is basically identical and the only difference is training sets?

mgrankin commented 3 years ago

I've taken a quick look at gpt2-simple. It seems they use SGD and gradient checkpoints to fit the large model. I'd suggest you to set the batch_size to 1, optim to SGD and fp16 with O2 and see how it goes.

fen0s commented 3 years ago

UPDATE: It did work! O2 arg for fpt16 finally fixed the not enough memory error. But now there's another issue that I have no idea how arises. Honestly, the whole dataset loading seems faulty, for some reason, considering it's second error with it. Screenshot_2 @king-menin, mind helping to figure out what it is? UPDATE 2: The features file generated seems to contain something really weird with seemingly broken encoding. Maybe that's the problem?

fen0s commented 3 years ago

Update #3: The line_by_line allows you to start finetuning for a few iterations, but apparently, on some line it stops. Does it mean that some lines from the dataset just can't be loaded? Screenshot_3 Update #4: Yeah, seems like it. Dataset with russian horror stories worked, while dataset of articles from foundation didn't. Seems like we'll need a function to preproccess dataset, apparently...

fen0s commented 3 years ago

PROBLEM SOLVED: GPT-2 model needs formatting. You can't just feed it raw text. You need to use <|n|> delimiter for your dataset, else it doesn't recognize it.. Honestly, you should update the pretraining section of readme, because this thing cost me 2 days of headache...