-
### Your current environment
```text
Collecting environment information...
PyTorch version: 2.1.2+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A
…
-
### Your current environment
The output of `python collect_env.py`
```text
# For security purposes, please feel free to check the contents of collect_env.py before running it.
python collect_e…
-
### Your current environment
> No module named 'vllm._version'
> from vllm.version import __version__ as VLLM_VERSION
> Collecting environment information...
> PyTorch version: 2.4.0+cu121
> Is…
-
### Your current environment
The output of `python collect_env.py`
```text
Collecting environment information...
PyTorch version: 2.4.0+cu121
Is debug build: False
CUDA used to build PyTorch…
-
### Your current environment
```text
Versions of relevant libraries:
[pip3] clip-anytorch==2.5.2
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.23.5
[pip3] onnx==1.14.0
[pip3] onnxruntime==1.1…
-
Hey! This is a super cool project.
I noticed your comment on https://news.ycombinator.com/item?id=38285191 about supporting open source models. I'm from the Anyscale team and we offer an OpenAI com…
-
Currently llamafiles are supported as generic OAI-compatible servers. While this works, it forces us to specify a valid (i.e. hf-transformers loadable) model name - which is needed for vLLM but might …
-
I am using the localGPT open source project to run the mistral 7b model on my RTX 3090 24gb GPU.
I can run the script with autogptq installed with pip, but get the following error:
xllamav2 kern…
-
## Description
Add typescript support to Node SDK.
## Reason
Currently the Node SDK has 0 type support and make developer experience tough when building an app with typescript and trying to use …
-
### Your current environment
```text
NB: This output is for our endpoint machine running vLLM
PyTorch version: 2.1.2+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to…