-
/vllm_2$ python examples/phi3v_example.py
WARNING 06-21 14:53:06 ray_utils.py:46] Failed to import Ray with ModuleNotFoundError("No module named 'ray'"). For multi-node inference, please install Ray …
-
### Describe the bug
I recently tried using openllm to connect to llama and it would give me some bentoml config errors. I'm not sure if its because I don't have a GPU but I didn't see any evidence o…
-
Model name: Lenovo Legion Slim 5 16APH8
CPU model: AMD Ryzen 7 7840HS
GPU model: NVIDIA RTX 4060 Mobile
Keyboard backlight: RGB
OS: Archlinux
Output of `sudo dmidecode -t system`. Please remove…
-
It seems that assisted generation can further reduce sampling latency. Is there scope for adding support for that in vllm?
Assisted generation [docs](https://huggingface.co/blog/assisted-generation…
-
### Your current environment
```text
PyTorch version: 2.3.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A
OS: Ubuntu 20.04.6 LTS (x86_64)
GCC ve…
-
### What happened?
Cannot Boot the image. Its not related to NVME or SDCARD boot but missing `.dtb` file
Process followed: [official](http://www.orangepi.org/orangepiwiki/index.php?title=Orange_Pi_3…
-
### Your current environment
```text
PyTorch version: 2.3.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A
OS: Ubuntu 22.04.4 LTS (x86_64)
GCC ve…
-
An idea that has been kicking around for years, but never written down:
The current definition of `int` (and correspondingly `uint`) is that it is either 32 or 64 bits. This causes a variety of pro…
-
multi image inference for "OpenGVLab/InternVL2-8B" not working
I got this inference code from here https://github.com/vllm-project/vllm/blob/main/examples/offline_inference_vision_language_multi_…
-
I tried using the `deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct` model and ran into this error:
`ValueError: Model type deepseek_v2 not supported.`
Any plans to support `deepseek_v2` soon?