-
### Your current environment
```
#!/bin/bash
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0
# Set default values
default_port=8008
default_model=$LLM_MODEL
defa…
-
### Your current environment
The output of `python collect_env.py`
```text
PyTorch version: 2.4.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A…
br3no updated
2 months ago
-
As described. The speculative decoding implementation is working, but should be sped up.
-
### Your current environment
Using the current official Docker image on Runpod with two 4090s:
```
alpindale/aphrodite-engine@sha256:b1e72201654a172e044a13d9346264a8b4e562dba8f3572bd92f013cf5420eb1…
-
**Environment:**
* WSL version: 2.2.4.0
* Kernel version: 5.15.153.1-2
* WSLg version: 1.0.61
* MSRDC version: 1.2.5326
* Direct3D version: 1.611.1-81528511
* DXCore version: 10.0.26091.1-2403…
-
Hi, @WoosukKwon and @zhuohan123 ,
Fantastic project!
I was taking a stab at implementing a version of **greedy** lookahead-decoding. Given some candidate completions, I was trying to:
1. Fork …
-
### Your current environment
The output of `python collect_env.py`
```text
Collecting environment information...
WARNING 09-21 15:29:13 _custom_ops.py:18] Failed to import from vllm._C with Im…
-
I tried to run the new version of Worker vLLM: `runpod/worker-v1-vllm:stable-cuda12.1.0`
> 2. Worker vLLM v1.1 with vLLM 0.5.3 now available under stable tags
> Update v1.1 is now available, use t…
-
### Your current environment
The output of `python collect_env.py`
WARNING 10-23 23:26:52 _custom_ops.py:19] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'")
…
-
### System Info
transformers==4.39.1
python==3.8.17
torch==2.0.1+cpu
### Who can help?
@sanchit-gandhi
### Information
- [ ] The official example scripts
- [ ] My own modified scr…