-
I'm having trouble interpreting some of the results...
After an Automatic Brute Search analysis, when I analyse the result_summary, I look at the Avegrage GPU Utilization.
How is this value de…
-
### Your current environment
vllm version: 0.5.4
gpu 24GB memory
### 🐛 Describe the bug
```bash
CUDA_VISIBLE_DEVICES=0 vllm serve mistralai/Mistral-7B-Instruct-v0.3 --api-key yyy --port 1…
-
I use this command to run docker `docker run --rm --gpus '"device=1"' --init --ulimit core=0 -p 8070:8070 -p 8081:8071 grobid/grobid:0.8.0` and then the GPU RAM consumed 20+ GB. But I found that and …
-
I just followed the tutorial to run the example code, but the model was running with less than 10% GPU usage. Why is this happening and can it be improved?
-
I noticed the api does not support processing batch prompt, the GPU utilization is low, and i want to use batch mode to improve GPU utilization and accelerate the inference process, so, how to do that
-
I can't seem to get any value that is not a `Binding` with the following code:
```
function GPUMonitor() {
const divide = ([total, free]) => free / total;
// nvidia-smi --query-gpu=timestamp…
-
In function [`get_max_gpu_blocks_available`](https://github.com/foundation-model-stack/fms-extras/blob/b1937db32f209166f07df9138f86ba72b14fa9f7/fms_extras/utils/cache/paged.py#L690)
The debugger sh…
-
Bug Description:
When the Triton Inference Server experiences high traffic, it appears to freeze and stops processing incoming requests. During this time, the GPU utilization reaches 100% and stays s…
-
### Anything you want to discuss about vllm.
I run the model on the server with 4 x NVIDIA GeForce RTX 4090 cards:
CUDA_VISIBLE_DEVICES=4,5,6,7 python -m vllm.entrypoints.openai.api_server --served…
-
### 🚀 The feature, motivation and pitch
I'm struggling to figure out how to extend our test suite to include vllm tests. The problem is that by default vllm will take over the whole gpu, which prev…