bug: Nemo model - Jan gets stuck on "Stopping model" after model settings change

mtomas7 commented 1 month ago

Jan version

0.56

Describe the Bug

Using imported GGUF model
Once Jan is closed and started again, Settings shows that model inactive, but once you want to start it, it is still stuck in the "Stopping" mode

Some ideas. In LM Studio, once you change the settings, there would appear a button, asking user to reload the model in order to pply the settings. In Jan you just start changing settings and it appears that app decides on its own to reload the model. That could be problematic, especially if user would be adjusting various settings. I would say - better go the LM Studio route.

Steps to Reproduce

No response

Screenshots / Logs

What is your OS?

[ ] MacOS
[X] Windows
[ ] Linux

app.log

Adding log file.

mtomas7 commented 1 month ago

Added log to the ticket.

mtomas7 commented 1 month ago

It looks I was able to figure it out. When I imported Nemo model, it automatically inherited the GPU settings: 33 layers, but because Nemo is larger, it could not fit and every time it would load, it would not able to complete the loading. After I manually decreased the # of layers, everything was good.

Perhaps it would be possible either to reset the GPU layer # or to re calculate, so the selected # of layers would actually fit into VRAM.

mtomas7 commented 1 month ago

There is still something wrong with Nemo - it loads, but once a string is passed, it doesn't generate response, but gives troubleshooting link. Also once model is loaded, GPU layer again are reset (to 44).

janhq / models