Out of Memory Error in Evaluation LLAMA

qiqiApink / MotionGPT

The official PyTorch implementation of the paper "MotionGPT: Finetuned LLMs are General-Purpose Motion Generators"

https://qiqiapink.github.io/MotionGPT/

198 stars 11 forks source link

Out of Memory Error in Evaluation LLAMA #4

Closed SHUWEI-HO closed 1 year ago

SHUWEI-HO commented 1 year ago

Hello, when I was executing evaluate llama with the loading of the llama model (7B), I encountered an out of memory error. I noticed that during the previous fine-tune training, I also had loading, but this issue did not occur. Could you please tell me what is the reason that makes this error happen?

qiqiApink commented 1 year ago

Maybe it is because the bfloat16 is used in training but not used in evaluation. You can refer to the training code to modify and try again.

SHUWEI-HO commented 1 year ago

Got it ! Thank you for your response.

bruinxiong commented 1 year ago

Got it ! Thank you for your response.

@SHUWEI-HO

how did you solve this issue ? I met the same problem. Thanks!

SHUWEI-HO commented 1 year ago

with fabric.device, lora(r=args.lora_r, alpha=args.lora_alpha, dropout=args.lora_dropout, enabled=True): config = LLaMAConfig.from_name(args.pretrained_llama) torch.set_default_tensor_type(torch.HalfTensor) model = LLaMA(config).bfloat16() torch.set_default_tensor_type(torch.FloatTensor)

gkuberreddy commented 11 months ago

What if my GPU doesn't support bfloat16() precision? How do I change and which lines to change to work with LoRA finetuning? Since I get the below error as shown in the image!