Open shahrukhx01 opened 10 months ago
accelerator_type:RTX: 0.01
for my RTX A6000, accelerator_type:H100: 0.01
for H100
SO I think you can try accelerator_type:RTX: 0.01
for your RTX 3090, too.
Hi @k0286, Thank you for your response. Did you mean accelerator_type_RTX: 0.01 for my RTX A6000, accelerator_type_H100: 0.01 for H100? Because adding :
after accelerator_type
might break YAML formatting.
Hi @k0286, Thank you for your response. Did you mean accelerator_type_RTX: 0.01 for my RTX A6000, accelerator_type_H100: 0.01 for H100? Because adding
:
afteraccelerator_type
might break YAML formatting.
I've also tried accelerator_type_RTX: 0.01
, but it doesn't work. After I trace the source code, I think accelerator_type:RTX: 0.01
is correct, and it work like a charm.
I am trying to deploy sharded LLMs to multiple RTX 3090s. So, far I have tried TGI by HF and it works fine. However, I came across Ray LLM at the last NLP summit and curious whether aviary support RTX series too. So far the pre-configured yaml files only point to
a100
,a10
, andv100
. Any leads to docs or sample configuration would be helpful. Thank you!