请问支持量化推理吗？

Meituan-AutoML / MobileVLM

Strong and Open Vision Language Assistant for Mobile Devices

Apache License 2.0

890 stars 64 forks source link

Closed LianghuiGuo closed 4 months ago

LianghuiGuo commented 4 months ago

可以指定load-8bit 或者 load-4bit吗？

er-muyue commented 4 months ago

Yes, please refer to here and turn on the load_8bit or load_4bit for your purpose.