search

Pan-ML / panml

PanML is a high level generative AI/ML development and analysis library designed for ease of use and fast experimentation.
MIT License
114 stars 16 forks source link

Idea: add option to load models with 8bit and 4bit quantization #46

Open Hanshan1988 opened 1 year ago

Hanshan1988 commented 1 year ago

Less memory requirement for 4bit and 8bit quantized models. https://huggingface.co/docs/transformers/main/main_classes/quantization

vsraptor commented 1 year ago

this ... :+1: