search

agemagician / ProtTrans

ProtTrans is providing state of the art pretrained language models for proteins. ProtTrans was trained on thousands of GPUs from Summit and hundreds of Google TPUs using Transformers Models.
Academic Free License v3.0
1.13k stars 153 forks source link

RuntimeError: CUDA out of memory. #79

Closed HanselYu closed 2 years ago

HanselYu commented 2 years ago

Thanks for your great work!

When I tried to run this code of Fine-tuning, that is ”ProtBert-BFD-FineTuning-PyTorchLightning-Localization.ipynb“ and '‘ProtBert-BFD-FineTuning-PyTorchLightning-MS.ipynb". I always got this error as follows: RuntimeError: CUDA out of memory. Tried to allocate 144.00 MiB (GPU 0; 9.78 GiB total capacity; 7.43 GiB already allocated; 124.31 MiB free; 7.54 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF Although my GPU has 10018MiB memory, it overflowed. And I found the memory consumption doubles with each step. I dont know why.

Thanks, Han

ratthachat commented 2 years ago

Hi, you can try if this updated version works for you? https://www.kaggle.com/code/ratthachat/prottrans-lightning-multitasks/notebook (It's not yet official, but @mheinzinger seems to accept this updated version; see this issue https://github.com/agemagician/ProtTrans/issues/74 )

HanselYu commented 2 years ago

Thanks for your provided sources. I would learn from it ! And I also succeed to run this code when I tried to swithch to the GPU with 24G memory.