search

BAAI-DCAI / Bunny

A family of lightweight multimodal models.
Apache License 2.0
866 stars 65 forks source link

请问支持量化推理吗？ #22

Closed LianghuiGuo closed 5 months ago

LianghuiGuo commented 5 months ago

请问可以支持量化加速推理么

YuanJianhao508 commented 5 months ago

Hey LianghuiGuo, 谢谢你的提问！Bunny现在支持8bit推理的，用bunny/serve/cli.py的--load-8bit就可以。4bit暂时还不支持。

LianghuiGuo commented 5 months ago

好像，感谢解答~