Currently, the Llama 2 model requires a significant number of GPUs to serve. Would it be possible to add support for quantized models to Ray-LLM? This would allow us to reduce the hardware requirements for serving Llama 2 models, making them more accessible to a wider range of users.
I have not been able to find any examples of quantized model serving for Ray-LLM, so it is not clear if this is currently supported.
Currently, the Llama 2 model requires a significant number of GPUs to serve. Would it be possible to add support for quantized models to Ray-LLM? This would allow us to reduce the hardware requirements for serving Llama 2 models, making them more accessible to a wider range of users.
I have not been able to find any examples of quantized model serving for Ray-LLM, so it is not clear if this is currently supported.