-
# Prerequisites
ROCm 6
# Expected Behavior
Attempting to utitilize llama_cpp_python in OobaBooga Webui
# Current Behavior
It loads the model into VRAM. Then upon trying to infer;
gml…
-
### What is the issue?
After the model is cleared from the graphics card RAM, when it is run again, the model is not loaded to the graphics card RAM but runs on the CPU instead, which slows it down a…
-
### What is the issue?
I have pulled a couple of LLMs via Ollama. When I run any LLM, the response is very slow – so much so that I can type faster than the responses I am getting.
My system speci…
-
### #
- [ ] I have searched the existing issues
### Current behavior
error log below
btw. same model and same mmproject-file works with koboldcpp , may you can copy paste ;)
### Minimum repro…
-
### Your current environment
The output of `python collect_env.py`
```text
Collecting environment information...
WARNING 08-27 02:59:41 cuda.py:22] You are using a deprecated `pynvml` packag…
-
## 🐛 Bug
After converting Mistral-Large-2407 and trying to load the model for chatting or serving, the following error is presented:
"(mlc-llm) USER@MBPM3MVLB ~ % mlc_llm serve /Users/USER/LLM/M…
-
### Your current environment
```text
(vllm) nd600@PC-7C610BFD7B:~$ python collect_env.py
Collecting environment information...
/home/nd600/miniconda3/envs/vllm/lib/python3.10/site-packages/torch…
-
### Is there an existing issue for this?
- [X] I have searched the existing issues
### Current Behavior
微调报错 ValueError: None is not in list
$ sh train.sh
08/16/2023 16:29:49 - WARNING - __main__…
-
### What happened?
I tried to run llama.cpp in Samsug Galaxy Tab S9 Ultra,the Android System is Android13.and I have compiled these libraries accoding the guide.I used these libraries in my APK and…
-
### Your current environment
```text
Collecting environment information...
PyTorch version: 2.3.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A
…