I'm running aviary (ray-llm) in the gpu instances rent from an online platform which only provides the running docker container. Therefore I can't directly run the "anyscale/aviary" image inside the instance but install all dependencies locally.
However, I have encountered the error as shown in the below image while the model is doing the warmup:
By checking the source code of text-generation-server, I found that no matter the normal model or flash_attention based model, their warmup functions only takes the batch parameter besides self. In the version <= 0.9.3, the warmup function takes parameters: (self, batch, max_total_tokens).
The warmup functions of v0.9.4 are shown below:
Change
Only one line change in /aviary/backend/llm/continuous/tgi/tgi_worker.py:
from
Context
I'm running aviary (ray-llm) in the gpu instances rent from an online platform which only provides the running docker container. Therefore I can't directly run the "anyscale/aviary" image inside the instance but install all dependencies locally.
The text-generation-inference I installed is the one cloned from this repo: https://github.com/Yard1/text-generation-inference/tree/main, the one listed in aviary README.md. The text-generation-inference version is 0.9.4.
However, I have encountered the error as shown in the below image while the model is doing the warmup:![Screenshot 2023-09-28 at 10 35 27 PM](https://github.com/ray-project/ray-llm/assets/13790266/200eaf1e-44e3-4ce4-ab83-ac2c3c3e41e5)
By checking the source code of text-generation-server, I found that no matter the normal model or flash_attention based model, their warmup functions only takes the
batch
parameter besidesself
. In the version <= 0.9.3, thewarmup
function takes parameters:(self, batch, max_total_tokens)
.The warmup functions of v0.9.4 are shown below:
![Screenshot 2023-09-29 at 12 03 07 AM](https://github.com/ray-project/ray-llm/assets/13790266/dcf03247-4330-4858-960d-f1b042767e92)
Change
Only one line change in
/aviary/backend/llm/continuous/tgi/tgi_worker.py
: fromto
Verification
Verified within my gpu instance with model
light-GPT
andllama2-13b-cht-hf