search

MAGICS-LAB / DNABERT_2

[ICLR 2024] DNABERT-2: Efficient Foundation Model and Benchmark for Multi-Species Genome
Apache License 2.0
212 stars 49 forks source link

I always encounter this error during the fine-tuning evaluation phase #78

Open zlw1747832053 opened 3 months ago

zlw1747832053 commented 3 months ago

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 4.44 GiB. GPU 0 has a total capacity of 10.58 GiB of which 4.44 GiB is free. Including non-PyTorch memory, this process has 6.14 GiB memory in use. Of the allocated memory 5.77 GiB is allocated by PyTorch, and 178.42 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

I always encounter this error during the fine-tuning evaluation phase.Even though I've drastically reduced the amount of data and batch size, I still report this error at pretty much the same time. As far as I can tell, memory usage starts to creep up at a certain node until it overflows, and I've tried a lot of things that don't work. My graphics card is 2080TI, CUDA 12.3.The strangest thing is that the memory is fixed during fine-tuning training, while it overflows during evaluation, even though the evaluation file is only 50MB

zlw1747832053 commented 3 months ago

I solved the problem for now by making the estimate data extremely small