search

dvmazur / mixtral-offloading

Run Mixtral-8x7B models in Colab or consumer desktops
MIT License
2.28k stars 223 forks source link

Mixtral OffLoading/GGUF/ExLlamaV2, which approach to use? #11

Open LeMoussel opened 6 months ago

LeMoussel commented 6 months ago

I'm a bit lost with the different quantization approaches such as GGUF, ExLlamaV2 & this project? Is it the same thing? Is one approach faster?

GGUF: TheBloke/Mixtral-8x7B-v0.1-GGUF ExLlamaV2: turboderp/Mixtral-8x7B-instruct-exl2

lavawolfiee commented 6 months ago

No, it's not the same thing.

Regarding ExLlamaV2 and llama.cpp (GGUF), I think it depends on your setup. As far as I know, ExLlamaV2 is faster on GPU but doesn't support CPU inference. llama.cpp on the other hand can split layers between CPU and GPU, reducing VRAM usage, and support pure CPU inference (initially, it was developed for CPU inference). They both are optimized for fast inference of LLMs and do their job pretty well. Note that they also have different quantization methods.

As for this project, we focus on optimizing inference for MoE-based models on consumer-class GPUs specifically. I can't tell you for sure right now when our method is faster/slower than the other ones, but we're currently researching that. It's also important to note that we used HQQ quantization, which is good but currently isn't very fast because it lacks good cuda kernels. Our team is actively working on supporting other quantization methods along with fast kernels and researching further possibilities to improve inference speed and quality.

Therefore, I believe our method is useful, at least if you don't have a lot of GPU VRAM (e.g., in Google Colab) or you want to fit a bigger model (with better quality) into it. We will do our best to implement new features and reach out to you as fast as possible.