-
## Description
I really love this project. I wanted to point out a few things, discuss them and eventually come up with some PRs.
First — I have WIP project called Réstral (Resume + Mistral), whic…
-
## Why BuckyOS?
Services running on the Cloud (Server) are closely related to our lives today, and people can hardly live without services in their daily lives. However, there is no operating sys…
-
## 🐛 Bug
I'm facing issue in serving Mistral-7B-Instruct-v0.3 model via mlc_llm serve. I get the below error when performing model serve with "python -m mlc_llm compile ..." command
INFO engine_…
-
Hey everyone,
Was doing some probing into why the group manager just fails to do its job and have some questions.
1) on lines 153 to 156, why are we broadcasting the message to all agents?
ht…
-
Im running a runpod serverless vLLM template with Llama 3 70B on 40GB GPU. One of the requests failed and I'm not completely sure what happened but the message asked me to open a github issue so I'll …
-
Although there are some lib wrappered vllm like TGI, but I want to know how to using vllm with stream output enabled, currently hard to found out-of-box example on it.
Typically, with original hf t…
-
Vulnerable Library - litellm-0.1.236-py3-none-any.whl
Library to easily interface with LLM API providers
Library home page: https://files.pythonhosted.org/packages/2e/1a/a6d54527bf31ececbe01e250eac9…
-
https://docs.vllm.ai/en/latest/models/lora.html
这是vllm框架多lora部署方案，想问下lmdeploy有类似的方法么
-
### Your current environment
```text
Collecting environment information...
PyTorch version: 2.3.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A
…
-
### Configuration
```hcl
terraform {
required_providers {
databricks = {
source = "databricks/databricks"
}
}
}
provider "databricks" {
profil…