search

GAIR-NLP / anole

Anole: An Open, Autoregressive and Native Multimodal Models for Interleaved Image-Text Generation
https://huggingface.co/spaces/ethanchern/Anole
618 stars 33 forks source link

Quantized Model #17

Open JoyBoy-Su opened 1 month ago

JoyBoy-Su commented 1 month ago

Speeding up Inference with Quantized models

dhruv1710 commented 1 month ago

Can I work on it? and we need to use dynamic quantization of pytorch right?

EthanC111 commented 1 month ago

Of course! Thanks a lot! You could submit a PR!

dhruv1710 commented 1 month ago

PR created #21

CoffeeShifter commented 1 month ago

What's the status of this? want to use quantized model on Windows

dhruv1710 commented 1 month ago

I am currently adding a feature so that you can use -q and you'll be able to use a quantised model for inference