su77ungr / CASALIOY

♾️ toolkit for air-gapped LLMs on consumer-grade hardware
Apache License 2.0
230 stars 31 forks source link

LlamaCpp change breaks Q4_0, Q4_1 and Q8_0 models #101

Closed su77ungr closed 1 year ago

su77ungr commented 1 year ago

Issue you'd like to raise.

Not really needed with our default q5 but it should be noted and added so it's within the doc

https://github.com/ggerganov/llama.cpp/pull/1508#issuecomment-1555031458

just a note to myself

Suggestion:

No response