-
### System Info / 系統信息
cuda 12.0 xprobe/xinference:v0.13.1 docker image from dockerhub
### Running Xinference with Docker? / 是否使用 Docker 运行 Xinfernece?
- [X] docker / docker
- [ ] pip install / 通过 …
-
swift infer一条条推理,真是受不了啊,得推到猴年马月……
vllm很多模型都不支持。
-
Running vllm according to instructions. Docker segfaults at startup, so I'm running straight on the machine.
Starting server with the following shell script. As you can see I've tried to turn max…
-
Going to give this a week to settle, there's always bugs when quants first land.
-
This machine has:
```
$ nvidia-ctk --quiet cdi list | grep -P nvidia.com/gpu='\d+'
nvidia.com/gpu=0
nvidia.com/gpu=1
nvidia.com/gpu=2
```
and I got:
```
File "/workspace/vllm/entrypoi…
-
### Your current environment
Failed to import from vllm._C with ImportError("/usr/lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /tmp/.conda/envs/vllm_env/lib/python3.10/…
-
### Describe the bug
Custom model name is not picked up.
WARNING:langfuse:Langfuse was not able to parse the LLM model. The LLM call will be recorded without model name. Please create an issue so we…
-
### Anything you want to discuss about vllm.
This document includes the features in vLLM's roadmap for Q3 2024. Please feel free to discuss and contribute, as this roadmap is shaped by the vLLM com…
-
### Anything you want to discuss about vllm.
```
(vllm) fm2024@fm2024:~/Micla/Project/vllm$ pip install -e .
Obtaining file:///home/fm2024/Micla/Project/vllm
Installing build dependencies ... do…
-
## Description
I would like to inquire if there are any plans to support more configuration settings for vLLM, specifically related to RoPE scaling and theta adjustments.
## Background
vLLM curre…