Closed sudujr closed 11 months ago
good question - in the future we will indeed support TPU and Inferentia; but for now this is probably not going to happen in the next 3 months. I'll close this issue for now and we can discuss more when that timeframe arises!
HI all,
I would love to know if in future will ray-llm support serving llm's on ML accelerators. With Vllms and other optimizations its possible to achieve high throughput and low latency. It would be great if we can use ray-llm's on ML accelerators as well.