Performence on terminal is very different within llamachat

[Darkhell101]

I find the performence when using llama.cpp in terminal is better in llamachat. The answer speed in terminal (interactive mode) is clearly faster on the same cpu usage level. It's a little strange. Maybe it can be solved.

[alexrozanski]

thanks @fsrmlgd -- one thing might be to try tweaking the number of CPU cores used for prediction in LlamaChat (in Settings > General) if this is different to llama.cpp

[Darkhell101]

Thanks for your reply. The app is very helpful. here is my test for two conditions.

https://user-images.githubusercontent.com/37462013/233781420-5c0dafc2-d201-4fca-b752-422c1b4d1b5b.mp4

https://user-images.githubusercontent.com/37462013/233781423-c425af30-640a-40b9-9987-9c5508e29f78.mov

As you can see the cat on top bar, It seems ridiculous. I change nothing about default settings.

[Darkhell101]

If you need, some models can be directly used on PC are listed here. https://huggingface.co/Pi3141 https://huggingface.co/eachadea So make one easy to find a suitable version.

[alexrozanski]

thanks @fsrmlgd, will take a look into this!

[spacecowgoesmoo]

I set up vicuna7b on an m1 Mac mini and am seeing the same thing. Text in llamachat generates very slowly just like in the video above. Mac was at 75% cpu usage.

(and this is fine if llamas are still too heavy for low end systems, but that doesn’t seem like the case here.)

[alexrozanski]

@spacecowgoesmoo thanks, still getting round to this!

[Freccia]

Hello, here to say I experience the same issue. On terminal it works smoothly, but the GUI makes it very slow. I am currently running on a Macbook Pro M1 with 16Gb of RAM.

btw amazing app design !