-
Similar to what was described here https://github.com/huggingface/candle/issues/2108
"When prompts get longer than trivial sizes, the memory usage spikes as the prompt is thrown into one Tensor and…
-
## 🐛 Bug
I tried to use Mistral Small 7B Instruct v0.3 as draft model for Mistral Large 2407. When not served using "--mode server", the model(s) never respond. I think that's because only CPU is u…
-
I am getting "float division by zero" error whenever I try to quantize mixtral related models with autogptq,
and here is my code.
```
from transformers import AutoTokenizer, TextGenerationPipeli…
-
Some open source models like Mistral-7b have 2 pytorch_model.bin files, BUT the order of the layers is changed
i.e
https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1/
The order of the l…
-
### System Info
Pre-built Docker image on g4dn.xlarge with Deep Learning OSS Nvidia Driver AMI GPU PyTorch 2.2.0 (Amazon Linux 2)
### Information
- [X] Docker
- [ ] The CLI directly
### T…
-
### Your current environment
The output of `python collect_env.py`
```text
PyTorch version: 2.3.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A…
-
Hello
I am a beginner-level user of PrivateGPT and set it for 'local' with LLM as mistral-7b-instruct-v0.2.Q4_K_M.gguf.
Please advise me how to add Groq (OpenAI compatible LLM service - https://…
-
I know this may not be supported given the added complexity, but I just wanted some clarification because I was surprised to see that I could pass these parameters to the server. Right now I non-deter…
-
Hey there,
I've seen several issues reported regarding the error mentioned above, so I wanted to share the fix I found.
**SPECS:**
- private-gpt version: 0.5.0
- LLM used: Mistral 7B Instruct …
-
here is the summary:
`unsloth/mistral-7b-v0.3-bnb-4bit` with error : ` KeyError: 'layers.0.mlp.down_proj.weight'`
`unsloth/Qwen2.5-7B-Instruct-bnb-4bit` with error: `KeyError: 'layers.0.mlp.down_pro…