-
Hi,
I am trying to set up vLLM Mixtral 8x7b on GCP. I have a VM with two A100 80GBs, and am using the following setup:
docker image: vllm/vllm-openai:v0.3.0
Model: mistralai/Mixtral-8x7B-Instruct…
-
With AMD APU (like my Ryzen 7940HX) it is possible to use "UMA" to extand VRAM. And in my case I can't alloc more than 4Go of VRAM (bios config).
And with this (https://github.com/ggerganov/llama.…
-
### Before submitting your bug report
- [X] I believe this is a bug. I'll try to join the [Continue Discord](https://discord.gg/NWtdYexhMs) for questions
- [X] I'm not able to find an [open issue](ht…
-
使用Extract Representations部分的命令：
```
sh scripts/save_reps.sh 0_forward belief
sh scripts/save_reps.sh 0_forward action
sh scripts/save_reps.sh 0_backward belief
```
Mistral-7B-Instruct-v0.2和deeps…
-
I'm trying to use the following as the model id and base name
MODEL_ID = "TheBloke/Mistral-7B-Instruct-v0.1-GPTQ"
MODEL_BASENAME = "wizardLM-7B-GPTQ-4bit.compat.no-act-order.safetensors"
But when…
-
Hello,
I'm currently working on a project that requires the use of the `TheBloke/Mistral-7B-Instruct-v0.1-GGUF` model, which is in the GGUF format. I've tried using the Hugging Face library to load…
-
### Your current environment
```text
The output of `python collect_env.py`
```
### 🐛 Describe the bug
ERROR 07-01 08:12:10 async_llm_engine.py:52] Engine background task failed
ERROR 07-01 08:…
-
### Question
Thank you for your excellent work. I am reproducing LLaVA with Mistral backened，but there is always a performance gap. Could you provide the training scripts (Pretrain and Finetune) for …
-
I am tring to reproduce some tasks' result on llava-1.6-mistral-7b, but found large gap on AI2D, ChartQA and InfoVQA. The lmms-eval version I ues is `0.2.0`.
My script:
```shell
python3 -m accele…
-
Hi,
I am trying to convert and save model: "mistralai/Mistral-7B-Instruct-V0.2" in 4bit and running into an error. I am using Flex GPU.
Could you please kindly help.
The error:
I am us…