-
- g4dn.2xlarge AWS instance
- CPU architecture x86_64
- CPU RAM: 32 GB
![image](https://github.com/NVIDIA/TensorRT-LLM/assets/12952636/e0b43d44-7a5f-46f7-92b5-32dbc79548a8)
- tensorrt_llm version:…
-
一些场景比如工单 bot 回复不需要实时回复,内存够的情况下,cpu 推理慢一点能出结果也可以
-
I follow the Doc: https://github.com/intel-analytics/ipex-llm/blob/main/docs/mddocs/DockerGuides/vllm_cpu_docker_quickstart.md
My Ecs is using Aliyun ecs.c8i.24xlarge ECS( https://help.aliyun.com/z…
-
With the ipex-llm docker container,
intelanalytics/ipex-llm-serving-vllm-xpu-experiment:2.1.0b2
it successfully load model in 4 ARC. But when load model in 8 ARC, it will have the following error…
-
### What is the issue?
Until v0.1.38 version my setup has 9 token/s but since the v0.1.39 to the actual v0.1.48 and v0.2 version it reduce the performance to 0.12 token/s
My setup:
- Intel(R) Cor…
-
### System Info
- CPU Architecture x86_64
- GPU: NVIDIA T4 * 4 (AWS g4dn.12xLarge)
- TensorRT-LLM v0.10.0
### Who can help?
[QiJune](https://github.com/QiJune)
@byshiue
### Information
…
-
**Description**
low speed in large concurrent requests
concurrent requests | 1 | 50 | 100
-- | -- | -- | --
TensorRT-llm| 73.36 | 193.30 | 193.81
Vllm| 64.13| 984.55| 1246.50
value is TPS …
-
When "pip install ipex-llm[cpp]", then "init-ollama.bat", it runs on CPU:
" ... msg="inference compute" id=0 library=cpu compute="" driver=0.0 name="" total="31.6 GiB" ... "
But when "pip install …
-
I have setup ipex-llm by following [install ipex-llm for llamacpp]( https://github.com/intel-analytics/ipex-llm/blob/main/docs/mddocs/Quickstart/llama_cpp_quickstart.md#1-install-ipex-llm-for-llamacpp…
-
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 30.45 GiB for max tokens in paged KV cache (99776).
[08/26/2024-14:34:24] [TRT-LLM] [I] Load engine takes: 26.76789355278015 sec
Input [Text 0]: "sy…