Closed Miracle2333 closed 8 months ago
when the low-resource inference is turned on for visual-chat, the inference speed decreased 5 times. Could you provide how to quantinize the model for speeding up the inference?
We are still working on developing the quantinized model.
when the low-resource inference is turned on for visual-chat, the inference speed decreased 5 times. Could you provide how to quantinize the model for speeding up the inference?