search

SciSharp / LLamaSharp

A C#/.NET library to run LLM (🦙LLaMA/LLaVA) on your local device efficiently.
https://scisharp.github.io/LLamaSharp
MIT License
2.65k stars 345 forks source link

[BUG]: gemma-2-9b-it-GGUF - error loading model #901

Closed nicholusi2021 closed 2 months ago

nicholusi2021 commented 2 months ago

Description

I've been using gemma-2-9b-it-GGUF in LM studio and it's working a lot better for me than Meta-Llama-3-8B-Instruct-GGUF. I really like it but I'm getting this error when I try to use it in LLamaSharp:   error loading model: error loading model architecture: unknown model architecture: 'gemma2'

I thought I had read that LLamaSharp now supports gemma2? I'm able to use LLamaSharp with Meta-Llama-3-8B-Instruct-GGUF just fine. Is it not supported yet, or maybe the model has an issue?

Thanks!

Reproduction Steps

To reproduce the error, try using gemma-2-9b-it-GGUF with LLamaSharp.

Environment & Configuration

Known Workarounds

No response

aezendc commented 2 months ago

I am having the same issue.

martindevans commented 2 months ago

I just tested gemma-2-9b-it-Q4_K_M and it works fine for me in one of the examples. Can you provide any more details about the error?

nicholusi2021 commented 2 months ago

hmmm, well I had downloaded the model manually from HuggingFace, but I think it was a bad model. When I just downloaded the LM Studio Community version, it worked for me. So I can confirm it works as long as it's a good model.