search

casper-hansen / AutoAWQ

AutoAWQ implements the AWQ algorithm for 4-bit quantization with a 2x speedup during inference. Documentation:
https://casper-hansen.github.io/AutoAWQ/
MIT License
1.58k stars 184 forks source link

DeepSeek-VL support #432

Open SinanAkkoyun opened 4 months ago

SinanAkkoyun commented 4 months ago

Hi! Thank you for your amazing work. I was wondering if DeepSeek-VL support would be possible? I'd love to serve a gemv_fast quant of it

(I think it is llama2 + the two vision encoders, but I believe one would copy HF inference code like for Llava)

SinanAkkoyun commented 4 months ago

I'd love to help if needed. I never implemented a model into an inference engine, if you are too busy to implement it yourself, could you please outline what I need to do?

SinanAkkoyun commented 4 months ago

Would you be interested?