mlc-ai / web-llm

High-performance In-browser LLM Inference Engine
https://webllm.mlc.ai
Apache License 2.0
13.11k stars 832 forks source link

Missing 32 Bit version of Mistral 7B? #444

Open flatsiedatsie opened 4 months ago

flatsiedatsie commented 4 months ago

In the current list of default models there is an odd ommission. While most models also have a 32 Bit variant available, Mistral 7B does not.

The practical result is that Linux users are missing out on performance, since their implementation of WegGPU currently only supports 32 Bit.

E.g.

Would it be possible to make sure all base models always have a 32 bit option available until Linux also supports 16 bit WebGPU?

flatsiedatsie commented 4 months ago

Also upgrading Mistral to the V3 version with 32K context and function calling support would still be wonderful.