lefnire
commented
1 year ago Some notes on tech.
Quantization. you'll see things like ggml-q5_0. GGML is the quantization technique or technology; there are others like GPTQ GGJT, etc. GGML was the preferred one when I researched, I hear there are newer/better techniques. Q means "quantized". A 32-bit floating precision weight in the model is rounded down to its 5 or 4-bit equivalent, which takes up that much less RAM & CPU. The last part _0, _1, etc - I don't know; I just think of it as like the fractional part, higher is better quality. Our target should be q5_0 if using GGML, as q5 is evidently much better quality (it's actually newer tech, they call q4 "legacy").
Parameters. 3b, 7b, 13b, etc. You only see good results 13b and up. Only in the 65b range are we competing with GPT-3.5. No models yet push GPT4, so GPT3 is the current benchmark. Take note of that! If you're ok on the OpenAI privacy stuff, you should just do it - it's cheaper, easier, and significantly higher quality. Ok, so if 65b is ideal quality, why go as low as 7b? It's because 3-13 is the range which can run on consumer hardware. Or even cloud hardware, if not parallelized across multiple machines. 13b is something of a consumer-hardware upper limit, 3b is too strong a quality sacrifice, so 7b is the sweet spot.
So: unless someone knows better with more recent models, ggml-7b-q5_0
[Edit 7/20/23]: Let's use Llama 2. AWS / Azure might have hosted versions too, so no local needed.
If there's any ticket I need engagement from the community, it's this one. Adding the ability for users to use their own locally-hosted Large Language Model like Llama, Vicuna, etc.
Current state: OpenAI
Going Premium unlocks OpenAI for (1) summaries & themes; (2) prompt. Summaries & themes without OpenAI are lower accuracy, based on pre-trained huggingface models less sophistication. But they shine for users who prefer isolated compute vs OpenAI. So it's a sort of trade: quality vs transparency. Prompt, on the other hand, is binary - you only get it if using Premium. One reason we charge for OpenAI usage is API costs, obviously. But another reason is the manual step someone goes through to put the site into OpenAI-mode, and there's messaging around what's about to happen, OpenAI's T&S, etc. So it's a gate. FWIW, we're looking into AWS's Titan, and Google's LLM API. AWS is disinterested in user data, so we'd prefer them - but haven't heard a peep for a while on Titan. Google on the other hand, while they have a history with privacy, they also have a history fixing said issues; eg with GDPR. It's all new territory, and we're keeping on top of it.
I want to note: @BirdieLady and I use and trust OpenAI (Prompt) with our journals. If you want my opinion, trust it. But also follow your heart. We find this feature almost as valuable as the whole site itself. It veritably doubled our value of Gnothi, with one single feature. There has been incredible personal depth exploration, new insights, and even actionable take-aways which have bettered our lives by using this feature. So it would be a travesty for it not to be used / explored due to the selection of backend API. So:
Ideal state: OpenAI or BYO LLM
For users who don't want to use OpenAI, but do want Prompt and the added quality of LLM summaries/themes; add the option to use one's own LLM. This would require:
We'll need to create a Wiki for setting all this up; for a list of recommended models, which models to avoid for wellness purposes, etc.
Hosting tools
Most of the action is happening here https://www.reddit.com/r/LocalLLaMA/. This user recommended exploring Kobold.cpp as a consistent/simple API setup on the users' machine:
Alternatives include llama.cpp and text-generation-web-ui. I've personally played with llama.cpp, it wasn't the level of simplicity I'd like to expect of our BYO-model users. There's also a project web-llm which would run the model in the user's browser, but that would require a very strong machine, and I fear it could more easily be misunderstood for its resource requirements (since it's so easy to "turn on") than if someone setup a BYO box at home. If anyone has experience / opinions here, please chime in!
Models
TL:DR: pick something from HF Leaderboard. Keep an eye on Meta's Llama v2.
As for the models themselves. Firstly, our best bet is anything quantized by TheBloke. His various GGML models take the original LLM and "minify" it via quantization to the point where a 7B or 13B parameter model, previously requiring cloud GPUs, can be run on local hardware. We should really be targeting 7B models, as that's the sweet-spot around 16-32gb RAM requirements; 13B models get very taxing.
When I left of my exploration (April 2023), I was poking around the list below. However, ignore this list and instead go to HF Leaderboard. That's constantly changing with SOTA. Also, Meta, whose first model Llama sparked the revolution but which has licensing constraints, will release a v2 which will likely clobber the leaderboard (and without licensing issues). My old list:
T5-Flan. Simple, terse. It's not a chat-bot, more a 1-word answerer. Very easily fine-tuned (trained), unlike the others (trains well with few samples, trains fast). See sample training, prompts, model, model docsBackground info:
Tasks
Consider web-llmNo-go, too intense requirement in-browser for little payoff. Good for fiddling, not for this.