parsehex
commented
2 months ago Need to write adapter for using koboldcpp conditionally.
IIRC the app is doing IPC to electron for SD and whisper, and API for llama.cpp
I think it would make sense to make an interface to unify the two sets. This would sit on the client end and be called to generate text, image, etc.
Maybe call it useAI. Also make a useKobold which useAI conditionally uses the appropriate composable depending on the platform. The rest of the client would use the useAI composable instead of others like useSD or useWhisper.
I recently bought a Steam Deck and found that it uses unified memory, allowing memory to be shared between the CPU and GPU. After digging into it, I found that there is about 9GB available to use as video memory by default, with the ability to increase to a total of 12GB thru the Deck's BIOS.
I was unsuccessful as getting a Vulkan build of llama.cpp to run, but I tried koboldcpp and it did run. This leads me to want to use llama.cpp/sd.cpp/whisper.cpp on Windows and koboldcpp for Mac and Linux. I've found that kobold's start-up time was slow on Windows but now on Linux, which is part of my rationale for having the difference.
Using koboldcpp would give us llama, sd and whisper in one binary, leaving us with only supplying the Piper binary which is simple since it's CPU only.
Supporting Steam Deck well will probably need a separate theme or UI and improving keyboard navigation is a must.
I've been able to use an AppImage on the Deck.