search

nomic-ai / gpt4all

GPT4All: Run Local LLMs on Any Device. Open-source and available for commercial use.
https://nomic.ai/gpt4all
MIT License
69.46k stars 7.61k forks source link

crash when launching Gemma-2-2b-it #2791

Open Turkur opened 1 month ago

Turkur commented 1 month ago

when trying to run models downloaded not from gpt4all the application crashes. The required models were downloaded to the required folder

Turkur commented 1 month ago

I tried running this and this https://huggingface.co/bartowski/gemma-2-2b-it-abliterated-GGUF https://huggingface.co/unsloth/gemma-2-it-GGUF/blob/main/gemma-2-2b-it.q8_0.gguf

ThiloteE commented 1 month ago

gemma2-2b is not yet supported in GPT4All 3.1. Requires newer llama.cpp. Meanwhile, you can try qwen2-1.5b-instruct, which is an llm almost similar in size, and great at benchmarks too.

ThiloteE commented 1 month ago

See https://github.com/nomic-ai/gpt4all/issues/2780

ThiloteE commented 1 month ago

Good to know. Sure, the Qwen2-1.5-Instuct is not the best model, but hey it's 1.5b. So far, I've had best results by using it for testing, because it is so small and also for RAG with documents that were mostly pre-formated in a Question / Answer style, which it was surprisingly pretty good at for it's size. I have not tested Gemma2-2b, so I cannot say anything about it.

GPT4All so far has a release cyclye that takes its fair time incorporating the newest llama.cpp. Since the release cycle is slower than some other apps, it is more stable, but the disadvantage is of course that, if newer models and features drop right after a release, it will take a while until it is supported in GPT4All. Just have a little patience.

On another note: Are you the Phil from huggingface? The one that criticized lots of model authors for failing to create models that do not hallucinate?

Phil209 commented 1 month ago

@ThiloteE Oops, I deleted my old comment just before you replied, so to provide context I claimed Gemma2 2b performs better than Qwen2 1.5b, particularly at instruction following and total general knowledge (~20 points high for both), making it a surprisingly functional general purpose LLM for its size, and far better than Qwen2 1.5b which can't even write poems that rhyme or stories that don't contradict a short list of simple prompt directives.

And yes, I'm the same phil that was on HF.

ThiloteE commented 1 month ago

Yes, I just checked benchmarks again and apparently Gemma-2-2b-it is higher there. So I am looking forward to it :D image

@Phil209 are you on the GPT4All discord too? I would love to have a conversation with you about model benchmarking and hallucinations :D

Phil209 commented 1 month ago

@ThiloteE Sounds good, see you on discord if they ever let me in without giving up my phone number.

Also, I wouldn't want to oversell Gemma2 2b. It's surprisingly good for a tiny LLM, but its LMsys rank (above GPT3.5 & Mixtral 8x7b) is completely detached from reality.

Turkur commented 1 month ago

gemma2-2b пока не поддерживается в GPT4All 3.1. Требует более новый llama.cpp. Между тем, вы можете попробовать qwen2-1.5b-instruct, который является llm почти такого же размера и также отлично подходит для бенчмарков.

Hello again. gpt4all has been updated but gemma 2 2b still doesn't work(

cosmic-snow commented 1 month ago

It is not yet supported.

The next release will probably be to address a bug with CUDA, not sure if anything else. Maybe the one after that will have support. No promises, though.