-
### Your current environment
Collecting environment information...
PyTorch version: 1.12.1+cu113
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A
OS: Ubun…
-
I'm following the code flow by running example.py and defining search as below,
`searcher = scann.ScannBuilder(normalized_dataset, 10, "dot_product").tree(
num_leaves=2000, num_leaves_to_search=…
-
### Proposal to improve performance
I've been trying to write a reliable benchmark to be used with vllm, and I discovered that when I use the openAI client it can't scale. If I try to use 50 concur…
-
There are several experiments being done with this repo to understand and evaluate the effects of quantization on the `llama2.c` models.
It is a great test-bed to analyze the effects of varying app…
-
Below are the PyTorch operators that are yet to be supported.
## Unclassfied (New)
N/A
## Primitives (Python operators)
- [x] aten::len
## Very easy (Constant generation or aliasing)
- [x]…
-
Hello,
1. The current implementation for matrix multiplication uses BRGEMM algorithm. Is there any implementation of "Low Rank Approximation approach" for matrix multiplication in oneDNN? Is there a…
-
硬件环境:4090+i9-14900f
操作系统:ubuntu 22.04
环境:python 3.8,vllm 0.5.5,vllm-flash-attn 2.6.1,transformers 4.45.0.dev0
问题描述:使用conda创建python3.8的环境后使用pip安装了vllm,模型权重文件也下载到了本地。然后执行启动命令”python -m vllm.entrypoin…
-
### Anything you want to discuss about vllm.
This document includes the features in vLLM's roadmap for Q3 2024. Please feel free to discuss and contribute, as this roadmap is shaped by the vLLM com…
-
Hi,
I have trained zipformer2 (without streaming) model with my dataset.
Training command: **./zipformer/train.py --num-epochs 40 --start-epoch 1 --use-fp16 1 --enable-musan False --exp-dir zip…
-
Prerequest
- [x] Introduce data type for int8 quantization
NYI cpu backend kernel: int8 quantization type
- [x] Add #5509
- [x] ArgMax
- [x] ArgMin
- [x] AvgPool2D #5593
- [ ] BatchMatmul
- …