vicuna 7b oom - Githubissues

SafeAILab / EAGLE

Official Implementation of EAGLE-1 (ICML'24) and EAGLE-2 (EMNLP'24)

https://arxiv.org/pdf/2406.16858

Apache License 2.0

842 stars 87 forks source link

Closed yjdy closed 9 months ago

yjdy commented 9 months ago

I'm trying to retrain the autoregression head with your train code on vicuna 7b I have 8 v100 with 32G. But even I set bs=1, it still out of memory.

My environment is as follow: CUDA 11.7 python 3.10 pytorch 2.1.2 transformers 4.37.2 accelerate 0.27.2

By the way, I used code of bsne1 branch

Best regards

liuy-2019 commented 3 months ago

how do you solve it ?