-
vLLM has a cousin named [aphrodite](https://github.com/PygmalionAI/aphrodite-engine) which adds EXL2, GGUF and a bunch of other handy features to it.
The good news is that the wheels already work o…
-
### Your current environment
```
%cd /kaggle/
!git clone -b rc_054 https://github.com/PygmalionAI/aphrodite-engine.git
%cd /kaggle/aphrodite-engine
!pip install -e .
```
### 🐛 Describe the bug
…
-
### Your current environment
```text
The output of `python env.py`
Collecting environment information...
PyTorch version: 2.3.0
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used …
-
### Your current environment
```python env.py
Collecting environment information...
PyTorch version: 2.3.0
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/…
-
### Your current environment
```text
PyTorch version: 2.3.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A
OS: Manjaro Linux (x86_64)
GCC version: …
-
### Your current environment
I have server with 4x3090ti. I can run llama 3 70b with vllm in docker with command:
`sudo docker run --shm-size=32g --log-opt max-size=10m --log-opt max-file=1 --rm -it…
-
### Your current environment
```text
PyTorch version: 2.3.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A
OS: Ubuntu 22.10 (x86_64)
GCC version: (…
-
### Your current environment
```text
PyTorch version: 2.3.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A
OS: Ubuntu 22.04.3 LTS (x86_64)
GCC vers…
-
### Your current environment
```text
/usr/lib/python3/dist-packages/scipy/__init__.py:146: UserWarning: A NumPy version >=1.17.3 and =1.17.3 and
-
### 🚀 The feature, motivation and pitch
Majority of the bots repeat sentences and due to that, it's annoying.
Implementing DRY sampler for Aphrodite Engine.
DRY is a sampler that forces the mod…