search

ray-project / ray-llm

RayLLM - LLMs on Ray
https://aviary.anyscale.com
Apache License 2.0
1.2k stars 87 forks source link

Release 0.5.0 #111

Closed sihanwang41 closed 6 months ago

akshay-anyscale commented 6 months ago

LGTM, let's have @avnishn or @Yard1 confirm

rifkybujana commented 6 months ago

How do i install tensorrt-llm in the ray-llm docker? I've tried to install tensorrt-llm several time but always stuck in the errors related to MPI or mpi4py

sihanwang41 commented 6 months ago

How do i install tensorrt-llm in the ray-llm docker? I've tried to install tensorrt-llm several time but always stuck in the errors related to MPI or mpi4py

Hi @rifkybujana, the trtllm will be installed by default inside the 0.5.0 image. Currently you can't install by yourself, we are planning to relax this restriction in the later version of rayllm.

rifkybujana commented 6 months ago

Hi @sihanwang41, thanks for the reply. When will the 0.5.0 image be released? Also which version of TensorRT-LLM would be installed?

sihanwang41 commented 6 months ago

Hi @sihanwang41, thanks for the reply. When will the 0.5.0 image be released? Also which version of TensorRT-LLM would be installed?

it will be out this week, 0.6.1 version will be installed.