Open CodeWithOz opened 3 weeks ago
UPDATE: I tried using an A100 single GPU with 40GB GPU memory and the same error happened. Seems there's a memory leak somewhere because the process just used up all the available memory. Here's the updated error message:
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 4.00 GiB. GPU 0 has a total capacity of 39.39 GiB of which 1.33 GiB is free. Process 35938 has 38.05 GiB memory in use. Of the allocated memory 35.03 GiB is allocated by PyTorch, and 2.21 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
I'm renting my GPUs from brev.dev by the way.
Is this issue resolved? I kind of encounter exactly same issue.
@CodeWithOz @zazabap Was the GPU running only the training? Can you provide the command you used to run the training? Additionally, what libraries and versions are you using?
I keep getting "CUDA out of memory" during training when finetuning Mistral 7B. My hardware is an NVIDIA A10G single GPU with 24GB GPU memory. The error message looks like this:
The value of reserved/unallocated memory ranges between 82MB and 1.08GB, so the error appears to happen both when there's enough and not enough memory.
I've tried the following measures with no success:
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
as recommended by the error messagemax_steps
from the default 300 to 200PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:512
(from here)max_steps
down to first 20, then 16, and ultimately 1torch.cuda.empty_cache()
before triggering the pytorch run{"messages": [{"role": "system", "content": "You are a helpful assistant."},{"role": "user", "content": "What is the capital of Portugal?"},{"role": "assistant", "content": "Lisbon"}]}
.seq_len
to 5000 following this commentThe README says best results require an A100 or H100, but single GPU machines can work with Mistral 7B. Given that I've tried to minimize so many parameters, is it really the case that bigger hardware is the only way forward?