search

ray-project / ray-llm

RayLLM - LLMs on Ray
https://aviary.anyscale.com
Apache License 2.0
1.2k stars 87 forks source link

Issue when running ray-llm with tensorrt-llm #139

Open leon-g-xu opened 4 months ago

leon-g-xu commented 4 months ago

I am seeing the following issue when using tensorrt-llm(0.6.1) on top of ray-llm (0.5.0) ImportError: cannot import name 'trt_llm_engine_py' from 'tensorrt_llm.libs' (unknown location) And looks like it's from this line of code: https://github.com/ray-project/ray-llm/blob/master/rayllm/backend/llm/trtllm/trtllm_models.py#L15

Wondering if anyone has run into this issue before?

leon-g-xu commented 4 months ago

Tagging @sihanwang41 and @shrekris-anyscale for awareness as you are both the committers on the file.