search

arcee-ai / mergekit

Tools for merging pretrained large language models.
GNU Lesser General Public License v3.0
3.99k stars 346 forks source link

Add --load-in-4bit and --load-in-8bit for HF eval backend #332

Open cg123 opened 1 month ago

cg123 commented 1 month ago

Allows using bitsandbytes quantization in mergekit-evolve when a) not using vLLM and b) not using in-memory mode.