-
### Your current environment
```text
The output of `python collect_env.py`
```
```
Versions of relevant libraries:
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.23.5
[pip3] torch==2.0.1+cu11…
-
Now LLAMA 3.1 is out, but sadly it is not loadable with current text-generation-webui. I tried to update transformers lib which makes the model loadable, but I further get an error when trying to use …
-
### What are you trying to do?
I want to know if my computer can support the model or not, but currently no one can tell me.
### How should we solve this?
Add the memory needed for each model tag i…
-
When trying to run the talk-llama example code with OpenCL enabled using a NVIDIA GeForce GT 755M, I get the following crash:
```
% LC_ALL=C ./obj-x86_64-linux-gnu/bin/talk-llama -mw ../nb-large-g…
-
Couqi Engine takes brakes mid sentence to load. IT takes sometimes between words or even in the middle of say the word. I tried to adjust setting but nothing works. I use i7 10th and RTX3060 computer.
-
### Bug report info
```plain text
➜ llm_playground git:(main) act --bug-report
act version: 0.2.60
GOOS: darwin
GOARCH: arm64
NumCPU: …
-
按照这个代码来看,多gpu、fastllm和量化不能同时使用吗
```
def get_model(args):
if not args.cpu:
if torch.cuda.is_available():
device = f"cuda:{args.gpu}"
elif torch.backends.mps.is_bui…
-
### Your current environment
Libraries Installed -
```
"vllm==0.5.5",
"torch==2.4.0",
"transformers==4.44.2",
"ray",
"hf-transfer",
"huggingface_hub"
```
### How would you like to u…
-
**LocalAI version:**
v2.4.1
**Environment, CPU architecture, OS, and Version:**
MBP 14 M1 PRO
**Describe the bug**
Not working make build and make BUILD_TYPE=metal build
**To Reproduce…
-
Hi,
I'm trying to run a rayllm as the tutorial in README.
But now my serving seemed to stuck at replica. It looked like this:
![image](https://github.com/ray-project/ray-llm/assets/101038773/…