-
### Your current environment
```text
Collecting environment information...
WARNING 07-23 19:11:42 _custom_ops.py:14] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm.…
-
I've done the following:
> Alternatively, one may also skip the quantization process and directy download the quantized VILA-1.5 checkpoints from [here](https://huggingface.co/Efficient-Large-Model…
-
I searched the outdated issues, and everyone is saying that the version of multi lora's punica must be >=8.0. Therefore, I want to ask if there is an option that only uses a standalone lora but suppor…
-
1. The scripts in `benchmarks` are only for latency benchmark. As we can compare it with other LLM inference framworks, is there any demo for offline throughput? I have compiled llm engine with option…
-
# Description of the new feature/enhancement
The Windows Terminal Chat currently only supports Azure OpenAI Service. This restriction limits developers who work with or are developing their own…
-
```python
import json
import ray
from ray.util.scheduling_strategies import PlacementGroupSchedulingStrategy
from vllm import LLM
from typing import Dict, List, Any
import numpy as np
from tqdm…
-
### Your current environment
```text
PyTorch version: 2.3.1+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A
OS: Ubuntu 20.04.6 LTS (x86_64)
GCC ve…
-
How to load multiple Lora weights and multiple text inputs to inference?
Currently, only single Lora weights and input tokens are supported as inputs. How to support multiple Lora weights and input t…
-
**Is your feature request related to a problem? Please describe.**
not everyone has access to openAI API. and even if you do have access, you're always limited by your budget. having an agent that ca…
-
### 🚀 The feature, motivation and pitch
Can the vllm acceleration framework be compatible with tensorRT-LLM?
here is doc about tensorRT-LLM:
https://github.com/NVIDIA/TensorRT-LLM
### Alternatives…