Closed qxj closed 11 months ago
我也有同样的问题 运行日志如下
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.95 GiB (GPU 0; 23.69 GiB total capacity; 18.23 GiB already allocated; 3.33 GiB free; 18.28 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
能够完成pretrain,但是在sft过程中会出现显卡内存溢出
我也有同样的问题 运行日志如下
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.95 GiB (GPU 0; 23.69 GiB total capacity; 18.23 GiB already allocated; 3.33 GiB free; 18.28 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
能够完成pretrain,但是在sft过程中会出现显卡内存溢出
和我情况一样哎,@DLLXW 帮忙看看?
我也有同样的问题 运行日志如下
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.95 GiB (GPU 0; 23.69 GiB total capacity; 18.23 GiB already allocated; 3.33 GiB free; 18.28 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
能够完成pretrain,但是在sft过程中会出现显卡内存溢出
和我情况一样哎,@DLLXW 帮忙看看?
我后面试了一下,把sft的batchsize调小就好了
显存溢出,调小batch就好,看你日志还有个无法调用Flash Attention的warning,需要升级下PyTorch到2.0,训练速度会快些,显存占用也稍微小些
显存溢出,调小batch就好,看你日志还有个无法调用Flash Attention的warning,需要升级下PyTorch到2.0,训练速度会快些,显存占用也稍微小些
谢谢,后来的确调小batch解决了。不过pytorch2.0训练nan一直没查出原因,所以还在用1.x https://github.com/DLLXW/baby-llama2-chinese/issues/17#issuecomment-1706116677
我刚刚也遇到了,微调昨天的预训练模型,发现全是nan,看了下做昨天的预训练日志后面也全是nan,过早重跑,结果日志还被覆盖了,目前还没复现出在哪出现的
运行日志: