Vision interactive mode for gguf models

EricLBuehler / mistral.rs

Blazingly fast LLM inference.

MIT License

4.49k stars 314 forks source link

Vision interactive mode for gguf models #882

Open ShaheerANative opened 4 weeks ago

ShaheerANative commented 4 weeks ago

Hi,

I am trying to load a Phi-3.5-3.8B-vision-instruct-Q8_0 GGUF model using the command for loading local GGUF file:

./mistralrs-server -i gguf --quantized-model-id path/to/files --quantized-filename path/to/model.gguf

^^ This command works fine for loading text-only interactive mode but adding vision-plain after the -i flag does not work to start image -based interactive mode.

Would be very useful to test GGUF vision models in this way, and could not find existing method to do so. Thanks.

DenisBobrovskiy commented 4 weeks ago

I am also trying to load a Phi-3.5 vision model through a gguf file, would appreciate any guidance on this, thanks!