-
### Proposal to improve performance
Test new feature medusa speculative sampling with [vllm v0.5.2](vllm-openai:v0.5.2).
After using Medusa speculative sampling, the performance dropped significantl…
-
- [ ] [cohereai_classify table | CohereAI plugin | Steampipe Hub](https://hub.steampipe.io/plugins/mr-destructive/cohereai/tables/cohereai_classify)
# TITLE: cohereai_classify table | CohereAI plugi…
-
### Your current environment
Collecting environment information...
PyTorch version: 2.3.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A
OS: Ubun…
-
Hi all
I am trying to use the LM Mistral in the dspy library with the following code:
```
llm = dspy.Mistral(
model="mistral-large-latest",
api_key="secret",
)
```
However, I enco…
-
# Prerequisites
Please answer the following questions for yourself before submitting an issue.
- [ X ] I am running the latest code. Development is very rapid so there are no tagged versions as …
-
Right now `/v1` is part of ENDPOINT.
But it is [more common to have BASE_URL](https://github.com/openai/openai-python?tab=readme-ov-file#configuring-the-http-client) (or API_BASE) setting, which incl…
-
## ❓ General Questions
When checking the Android demo app and Build Android App from Source, only 'arm64-v8a' is supported.
I wonder if it is possible to build the 'mlc_llm package' a and th…
-
# Prerequisites
Please answer the following questions for yourself before submitting an issue.
- [x] I am running the latest code. Development is very rapid so there are no tagged versions as of…
nih23 updated
6 months ago
-
### Your current environment
```text
Collecting environment information...
PyTorch version: 2.4.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A
…
-
### Your current environment
The output of `python collect_env.py`
```text
No module named 'vllm._version'
from vllm.version import __version__ as VLLM_VERSION
PyTorch version: 2.4.0+cu121
…