-
**Is your feature request related to a problem? Please describe.**
Support sqlite database as batch data source for the offline store.
**Describe the solution you'd like**
SQLite as an offline st…
-
### Your current environment
```text
python3 collect_env.py
Collecting environment information...
PyTorch version: 2.3.1+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM use…
-
Greate work, how to use llama 2 as LLM engine? Thanks.
-
as title.
thanks
-
Hello, FlexFlow team!
Thank you for your outstanding work! I am attempting to reproduce the experimental results from the paper "SpecInfer: Accelerating Generative Large Language Model Serving with…
-
Offline LLMs + online browsing if available is a use case for private agents.
GPT4All.io has an easy installer and runs on CPU on most PCs.
Vicuna https://vicuna.lmsys.org - GPT-4 with ~90% Chat…
vRobM updated
7 months ago
-
运行ChatGLM-6B报错误后(错误信息见issue-运行tutorial中的ChatGLM-6B报grpc错误),发现VGPU-CORE资源不足,但是eggroll的dashboard展示的可分配VGPU-CORE资源数量是正常的。
到mysql中手动修改node 和processor manage表,将deepspeed任务pre-allocated的VGPU-CORE记录清除,才能重新提…
-
I would like to request a feature enhancement for the vectra CLI functionality. Specifically, I would like to have the option to use local Large Language Models (LLMs) instead of relying solely on Ope…
-
1. offline serving
![image](https://github.com/vllm-project/vllm/assets/43260218/87e216b5-9064-4c2a-a021-cac08e22795d)
2. online serving(fastapi)
![image](https://github.com/vllm-project/vllm/ass…
-
### Question Validation
- [X] I have searched both the documentation and discord for an answer.
### Question
I am concerned about data privacy. For example, if we are using paid LLMs via an API lik…