search

rustformers / llm

[Unmaintained, see README] An ecosystem of Rust libraries for working with large language models
https://docs.rs/llm/latest/llm/
Apache License 2.0
6.06k stars 350 forks source link

WizardCoder llama assert failure #417

Open jacohend opened 10 months ago

jacohend commented 10 months ago

Trying to run a variety of ggml models from TheBloke leads to this error: GGML_ASSERT: llama-cpp/ggml.c:6270: ggml_nelements(a) == ne0*ne1*ne2

Wondering if anyone else is experiencing this, and what the issue might be?

jacohend commented 10 months ago

Related: https://github.com/ggerganov/llama.cpp/issues/2445#issuecomment-1656949448

LLukas22 commented 10 months ago

Probably another issue with the currently used ggml version, a re-sync with the current main branch of llama.cpp is probably needed.

jacohend commented 10 months ago

I actually did that and found a failure on the same assert line. The linked comment said rolling the version back worked best.

I'm wondering if this assert is assuming constant layer sizes, so any modification like The Bloke does might be causing the failure?