4_0, 4_1, and 8_0 Quantized models fail to load - SIGSEGV invalid memory reference

[sigaloid]

I've quantized all of my models with the quantize-all script and this led me to find out a large portion of quantized models don't load properly in whisper-rs. Any model quantized at q4_0, q4_1, or q8_0 do not load and throw a SIGSEGV invalid memory reference. Oddly, ggml-large-v1-q4_0.bin throws a proper error but the rest of them (that I tested) core-dumped.

These work fine in whisper.cpp when running in ./main.

Attached is a ggml-tiny-q4_0.bin that fails to load with the error. Quantized with

./quantize models/ggml-tiny.bin models/ggml-tiny-q4_0.bin q4_0

ggml-tiny-q4_0.bin.zip

[tazz4843]

I actually hit this exact issue, and found the source of it: ggerganov/whisper.cpp#993

A workaround for now is cloning this repo, then updating the git submodule of whisper.cpp to point to the latest upstream instead of the current release tag it's fixed at.

[sigaloid]

Ok, great to hear that it's patched upstream. thanks!

[tazz4843]

See #85