-
### System Info
CPU: X86
Memory size: 2TB
GPU Name: H20
TensorRT-LLM: 0.10.0
OS：Alibaba Cloud Linux release 3 (Soaring Falcon)
GPU Driver：550.54.15
CUDA：cuda_12.4.r12.4/compiler.33961263_0
Do…
-
I recently reinstalled comfyui and opened a previously used .json. suddenly there are two nodes inside of searge llm node although the code does not include Searge_AdvOptionsNode. basically the node i…
-
### Is there an existing issue for the same bug?
- [X] I have checked the existing issues.
### Describe the bug and reproduction steps
1 - Follow installations steps ...
2 - Get error msg `SANDBOX…
-
### Proposal to improve performance
I am trying to run phi3.5 vision instruct model with around 10k prompts. What I noticed with the increase in prompts my CPU RAM consumption keeps increasing and ev…
-
使用ollama运行minicpm-v模型，调用过程中发现，单独调用llm文字部分，正常运行到igpu。
但是同时使用图片和文字，会出先LLM运行到CPU上。
ollama run minicpm-v:latest
Test prompt
{
"model": "minicpm-v:latest",
"prompt": "图片讲了什么内容?",
"images":[…
-
Hi there, I would like a raise an issue about the future enhancement of `nano-graphrag`. One of the big direction is add more components to adapt various scenes.
What would you guys like to add? Fo…
-
Hello!
I see that vLLM got updated in the latest version of IPEX-LLM and so decided to try using it with Llama-3.2-11B-Vision, however I seem to get errors each time:
`2024-11-13 14:41:58,608 ER…
-
### System Info
- CPU architecture: x86_64
- GPU name: NVIDIA A100
- TensorRT-LLM tag: v0.10.0
### Who can help?
@byshiue
### Information
- [ ] The official example scripts
- [ ] My …
-
Currently have an LLM engine built on TensorRT-LLM. Trying to evaluate different setups and gains on types.
Was trying to deploy the llama model on a multi-gpu, whereby between the 4 GPUs, I would hav…
-
```
task_manager = TaskManager(self.agent_config.get("agent_name", self.agent_config.get("assistant_name")),
bolna-app-1 | File "/app/bolna/agent_manager/task_manager.py", line 58, in __init__
…