Closed Ichigo3766 closed 1 year ago
Facing the same Issue.
Because the VRAM is not released, after subsequent n requests the server crashes with out of memory for me.
Currently after every n requests, it crashes and i restart the docker and repeat the cycle.
I monitored with watch nvidia-smi and I can see the memory climbing up with each request till the point of crash and this process so repeats.
Here's the docker command I used
docker run --gpus all --shm-size 20g -p 8080:80 --name tiiuae/falcon-40b-instruct --log-driver=local --log-opt max-size=10m --log-opt max-file=3 -v $volume:/data --env BUILD_EXTENSIONS=False --env NCCL_SHM_DISABLE=1 ghcr.io/huggingface/text-generation-inference:0.9 --model-id tiiuae/falcon-40b-instruct --num-shard 4 --quantize "bitsandbytes" --max-best-of 1 --max-total-tokens 2048 --max-input-length 2047 --trust-remote-code
I would like to understand why the vram is not being released after a request completion?
It is!
Even though the nvidia-smi or other tools might not show it, PyTorch correctly destroy the tensors but this will not be displayed on nvidia-smi as PyTorch has its own allocator.
Basically when you create a new tensor on GPU, PyTorch allocates the memory on the device. When you destroy this tensor, PyTorch keep the memory space inside it's own cache allocator and doesn't release the device memory. This is done to speed up new allocations. Only calling torch.cuda.empty_cache releases the PyTorch cache allocator and display the true free memory in nvidia-smi.
We do this in some operations to avoid memory fragmentation after a long time but it is not required to do it after each completion.
See https://pytorch.org/docs/stable/notes/cuda.html#cuda-memory-management for more info on this subject.
@bikal-netomi, if you are not running on 0.9.1, can you try it? I think that it's better to open a new issue on this as this problem might only be related to the bitsandbytes integration.
Replying back as im still unsure about it because im facing this issue along with many others and its not just few gb but a whole 40-45gb worth of vram that is not being cleared and ends up causing the server to crash. I am not using any quantizing whatsoever.
Is there any suggestions that could solve this problem as i have tried lowering the tokens and everything and it just keeps filling the vram till it crashes unless i stop the connection mid stream which resets the whole vram back to normal. But as you would know this is not an efficient thing to do. Its causing issues as im having to restart the server multiple times a day due to the context load.
Thank you!
1. The full command line used that causes issues: docker run --gpus all --shm-size 40g -p 1000:80 -v $volume:/data hug:gptq --model-id TheBloke/WizardLM-13B-V1-1-SuperHOT-8K-fp16 --num-shard 4 --trust-remote-code --max-input-length 4096 --max-total-tokens 10000 --max-batch-total-tokens 10000
OS version: Amazon Linux 2
Model being used TheBloke/WizardLM-13B-V1-1-SuperHOT-8K-fp16
Hardware used (GPUs, how many, on which cloud) (nvidia-smi): 4 Nvidia A10G's totaling 96gb vram on AWS EC2 instance
Built the docker image locally pulling the most recent changes as of now
Now I will ask multiple queries and show below of how fast the vram goes up and stays there
It took 8 queries of around 2k ish tokens each to crash the server. The queries were sent one at a time. Note: While this memory is going up, I am not able to use the vram at all by anything else and the responses start slowing down as well.
2023-07-15T04:17:14.473703Z ERROR batch{batch_size=1}:prefill:prefill{id=45 size=1}:prefill{id=45 size=1}: text_generation_client: router/client/src/lib.rs:33: Server error: Allocation on device 3 would exceed allowed memory. (out of memory)
Whats interesting is how the last gpu memory gets dropped even though when the server crashes, all gpu's were maxed out and only the last gpu drops memory but still the server is unable to allocate.
Hopefully this helps!
Hi,
I encountering same issue starting with TGI 0.9+.
System Info OS version: Ubuntu 22.04.2 LTS Model:
{
"model_id": "tiiuae/falcon-40b",
"model_sha": "561820f7eef0cc56a31ea38af15ca1acb07fab5d",
"model_dtype": "torch.float16",
"model_device_type": "cuda",
"model_pipeline_tag": "text-generation",
"max_concurrent_requests": 128,
"max_best_of": 2,
"max_stop_sequences": 4,
"max_input_length": 1535,
"max_total_tokens": 2048,
"waiting_served_ratio": 1.2,
"max_batch_total_tokens": 16000,
"max_waiting_tokens": 20,
"validation_workers": 2,
"version": "0.9.1",
"sha": "31b36cca21fcd0e6b7db477a7545063e1b860156",
"docker_label": "sha-31b36cc"
}
Hardware used:
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.54.03 Driver Version: 535.54.03 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA A100-PCIE-40GB On | 00000000:5E:00.0 Off | 0 |
| N/A 33C P0 30W / 250W | 4MiB / 40960MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 1 NVIDIA H100 PCIe On | 00000000:86:00.0 Off | 0 |
| N/A 53C P0 86W / 350W | 81001MiB / 81559MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
| 2 NVIDIA H100 PCIe On | 00000000:D8:00.0 Off | 0 |
| N/A 48C P0 78W / 310W | 81001MiB / 81559MiB | 0% Default |
| | | Disabled |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 1 N/A N/A 9284 C /opt/conda/bin/python3.9 80988MiB |
| 2 N/A N/A 9285 C /opt/conda/bin/python3.9 80988MiB |
+---------------------------------------------------------------------------------------+
Runtime TGI Docker 0.9.1
Reproduction
docker run -it --rm --gpus '"device=1,2"' --shm-size 1g -p 10000:80 -v $(pwd)/data:/data -e USE_FLASH_ATTENTION=true ghcr.io/huggingface/text-generation-inference:0.9.1 --model-id tiiuae/falcon-40b --num-shard 2 --max-batch-total-tokens 16000 --max-input-length 1535 --max-total-tokens 2048
docker exec -it 2fc7d8dba690 text-generation-benchmark --tokenizer-name tiiuae/falcon-40b -b1 -b4 -b8 -b16 -b32 -b64 -b128 -b256 -b 384 -b512
Hope it helps!
Could you try ghcr.io/huggingface/text-generation-inference:sha-a2cf1bd and see if it solves your issue?
Hi,
Same problem here (sha-a2cf1bd) :
No problem with 0.8.2.
@OlivierDehaene Thank you so much! I believe that fixed the issue. I will leave this open for others to try it and then close if everything is good!
@OlivierDehaene I can confirm that issue is fixed on my side too. Thanks a lot!
That's still very weird. We shouldn't have to empty the cache manually like this. I will see if the torch folks have a better answer than this and report back to this issue.
@OlivierDehaene Thank you for the fix.
H!
I would like to understand why the vram is not being released after a request completion? So something I noticed is that when I send queries the vram gets filled but after the answer is received, that query still remains inside the vram.
Now whats interesting is that, lets say you query the model but rather than letting it finish the response, you close the connection mid response, all the vram gets flushed and we are only left with the model vram which is perfect. So why is that not happening automatically after each request but a connection close in the middle of the request clears all the vram?
I have tested this multiple times from the langchain wrapper, a flask website, curl and all are able to output the same result. The model loads fine and starts at around 40gb and then lets say you do long conversations, it will go all the way up to 90gb and stay there and not flush but the moment i do the trick of closing the connection, the vram resets right back to 40gb.