Macos -- App chat crash when context length set to 4096

reorproject / reor

Private & local AI personal knowledge management app.

https://reorproject.org

GNU Affero General Public License v3.0

6.56k stars 375 forks source link

Macos -- App chat crash when context length set to 4096 #26

Open pl2k2000 opened 5 months ago

pl2k2000 commented 5 months ago

I am on Macos M2 with 24GB Ram and loaded mixtral_7bx2_moe.Q8_0.gguf or guanaco-13b-uncensored.Q4_K_M.gguf

If I select context length to 4096 it will crash when open the chat window. Context length 2048 works fine. System showing more than 16GB memory avaliable when started Reor.

samlhuillier commented 5 months ago

Yes it uses more RAM when you initialise a model with more context. Wasn't 2048 enough for context?

pl2k2000 commented 5 months ago

My point is that I have enough Ram for the model plus 4096 context. I have no issue/crash when used other local LLM apps with 4096 context on the same model. My model is only 8GB in size and I have 24GB Ram. Tell me why it is not enough?

samlhuillier commented 5 months ago

Gotcha I'll take a look