-
### Your current environment
```text
The output of `python collect_env.py`
Collecting environment information...
PyTorch version: 2.3.1+cu121
Is debug build: False
CUDA used to build PyTorch: 12…
-
Hi,
Is there a way to use GGUF with BIGDL? For example, I would like to run llama-2-13b-chat.Q4_0.gguf.
Thx...sanjay
-
Hi I tried to run LLMLingua using dolphin-2.6-phi-2 but I got
```AssertionError: Torch not compiled with CUDA enabled```
```
PS C:\Users\DefaultUser> python "C:\Users\Public\Coding\LLMLingua\LLM…
-
### Describe the bug
python run_llama_int8.py -m ${MODEL_ID} --quantized-model-path "./saved_results/best_model.pt" --benchmark --jit --int8-bf16-mixed
prompt = ["Is Apple an American company?",…
-
### What are you trying to do?
I want to know if my computer can support the model or not, but currently no one can tell me.
### How should we solve this?
Add the memory needed for each model tag i…
-
I tried the inference by using BigDL LLM optimized Qwen-14B-Chat model in Aliyun (64 core, Intel(R) Xeon(R) Platinum 8369B, 256GB). For simple test case (one simeple question), there was a big perfoma…
-
**Describe the bug**
Using v0.2.146, installation works fine, but when i finished to create the integration, i got a "failed to configure" message.
**Expected behavior**
Integration shall s…
pbn42 updated
1 month ago
-
### What happened + What you expected to happen
### What Happened:
When deploying models using RayServe with autoscaling enabled on Amazon EKS, specifically across multiple `inf2` nodes, the syste…
-
### Your current environment
```text
The output of `python collect_env.py`
Collecting environment information...
WARNING 07-15 19:13:04 _custom_ops.py:14] Failed to import from vllm._C with Mo…
-
Is that supported? I mean S-LoRA mechanism.
If yes, is there any example for that?
If no, any future plans?