search

abdeladim-s / pyllamacpp

Python bindings for llama.cpp
https://abdeladim-s.github.io/pyllamacpp/
MIT License
60 stars 21 forks source link

It still does not work with some GGML v3 models #11

Open idobric opened 1 year ago

idobric commented 1 year ago

For example: https://huggingface.co/TheBloke/gpt4-alpaca-lora-30B-4bit-GGML/tree/main The main branch should have GGML v3 but python crashes when trying to run this model. I can run it using llama_cpp package.

Thanks

abdeladim-s commented 1 year ago

@idobric, it seems like there are new updates again :sweat_smile: the main branch is pointing to the GGML v2 I guess. I will update it to the latest and let you know.

abdeladim-s commented 1 year ago

@idobric, I updated llama.cpp to the latest release and pyllamacpp is working now with the model you provided above. I've pushed a new 2.4.0 version to PYPI. Could you please test it on your end ?

ParisNeo commented 1 year ago

Hi and thank you very much for doing this.

What's really hard about all these breaking changes is that they never keep retrocompatibility whith previous models versions.

Your bindings are supported at my gpt4all-ui. It would be cool if you test it and upgrade the binding part as you change yours your bindings has -a at the end of the name (for abdeladim) as i also support other developers bindings. Marella has also made some good bindings and i add -m to distinguish his from yours. I probably gonna change the bindings list to have a card that shows who built it and give a link to their repos to show some love.

I have created a binding template to show how to interface any new binding with the app. It is very easy, just implement a class and you're good to go.

I finally built a video to show how the basic use of the ui. I hope you like it: https://www.youtube.com/watch?v=ds_U0TDzbzI&ab_channel=ParisNeo

abdeladim-s commented 1 year ago

@ParisNeo, yeah, that's the big problem, it's hard to know which version for which model, especially with the myriad of ggml models out there.

So finally the 2.0 release of gpt4all-ui is here, it looks fantastic :clap:

Yeah, I noticed that you support my bindings. Glad you found them useful. I didn't change anything to the API so basically everything should still work I guess. If I did I'll try to submit a PR.

Keep up the good work :muscle:

ParisNeo commented 1 year ago

Thanks man :)