search

ray-project / ray-llm

RayLLM - LLMs on Ray
https://aviary.anyscale.com
Apache License 2.0
1.22k stars 87 forks source link

Ray LLM on Nvidia RTX series? #72

Open shahrukhx01 opened 10 months ago

shahrukhx01 commented 10 months ago

I am trying to deploy sharded LLMs to multiple RTX 3090s. So, far I have tried TGI by HF and it works fine. However, I came across Ray LLM at the last NLP summit and curious whether aviary support RTX series too. So far the pre-configured yaml files only point to a100, a10, and v100. Any leads to docs or sample configuration would be helpful. Thank you!

k0286 commented 10 months ago

accelerator_type:RTX: 0.01 for my RTX A6000, accelerator_type:H100: 0.01 for H100 SO I think you can try accelerator_type:RTX: 0.01 for your RTX 3090, too.

shahrukhx01 commented 10 months ago

Hi @k0286, Thank you for your response. Did you mean accelerator_type_RTX: 0.01 for my RTX A6000, accelerator_type_H100: 0.01 for H100? Because adding : after accelerator_type might break YAML formatting.

k0286 commented 10 months ago

Hi @k0286, Thank you for your response. Did you mean accelerator_type_RTX: 0.01 for my RTX A6000, accelerator_type_H100: 0.01 for H100? Because adding : after accelerator_type might break YAML formatting.

I've also tried accelerator_type_RTX: 0.01, but it doesn't work. After I trace the source code, I think accelerator_type:RTX: 0.01 is correct, and it work like a charm.