search

topal-team / rockmate

GNU General Public License v3.0
33 stars 5 forks source link

support for bitsandbyte quantized models #1

Open cerisara opened 1 year ago

cerisara commented 1 year ago

Hi,

Thanks for sharing your code, it's really great. I'm wondering whether your library supports 4-bit quantized (with bnb) GPT-like models?

Thanks!

Nagael commented 1 year ago

Hi @cerisara ,

Thanks for your interest in our software. We had not tested bitsandbytes yet, but from my first limited attempts it seems to be working correctly. Please let us know if you encounter any issue when using it.

Best.