-
Something interesting occurred while upgrading to version 1.8.0. Previously, it had been throwing an "Out of Memory" error, but that issue has now been resolved. However, a new problem has surfaced, w…
-
When running models using the cli the whole names needs to be used e.g. `ollama run deepseek-coder-v2`. Some of these names are hard to remember. I often copy them from `ollama list`. What if we could…
-
I have enabled gpt4all using env variables but I still get the window to configure an OpenAI API key (or custom).
I'm using the dev version because I want it running on the localhost only.
Is th…
-
having issues with autoAWQ not detecting my RTX 3070ti with Cuda 12.4 installed when attempting to run with loading "TheBloke/Mistral-7B-OpenOrca-AWQ"
the command nvidia-smi returns:
```
Sun Apr …
-
### 🚀 The feature
Requested by a user on Discord:
- Discord link: https://discord.com/channels/1121119078191480945/1125758905310519327/1195257282821369867
- Anyscale blog: https://www.anysc…
-
I am trying to create a chatbot using mistral 7b model (mistral-7b-openorca.Q4_K_M.gguf) . The model should reply "I don't know" for latest events questions like (What is the weather today in Delhi?, …
-
when I tried to run Mistral-7B-OpenOrca (using oobabooga/text-generation-webui)Ж
ImportError: libcudart.so.12: cannot open shared object file: No such file or directory
-
1. offline serving
![image](https://github.com/vllm-project/vllm/assets/43260218/87e216b5-9064-4c2a-a021-cac08e22795d)
2. online serving(fastapi)
![image](https://github.com/vllm-project/vllm/ass…
-
i am doing mistral 7b openorca inference using llamacpp-python but its is taken lot of timing.How can i fix that
llama-cpp-python version is 0.2.11
Server Configuration:
1)Windows Server 2022 Sta…
-
I am trying to run the summarization example as is - however that fails since the device of outlines.transformers is not set to cuda.
The example uses model "TheBloke/Mistral-7B-OpenOrca-AWQ" which r…