search

Macoron / gpt4all.unity

Bindings of gpt4all language models for Unity3d running on your local machine
MIT License
100 stars 16 forks source link

Support for falcon models #11

Open AntoineCharton opened 1 year ago

AntoineCharton commented 1 year ago

Hello :) Really nice repo!

https://docs.gpt4all.io/ Falcon transformer architecture seems to be supported by gpt4all is it possible to add it to the library?

Macoron commented 1 year ago

This repo uses relatively old version of gpt4all, which sadly doesn't support Falcon and some newer architectures. This could be fixed by updating compiled libraries and C# bindings.

I'll probably do this in the future, but if anyone want to do PR with update - I'll glad to merge it.

AntoineCharton commented 1 year ago

Ok will try this week-end. Anything you did in particular for the build that's not on the instructions?

Macoron commented 1 year ago

Not really. Check build_cpp.sh and build_cpp.bat for Mac and Windows builds.

Right now gpt4all.unity compiled using this commit: https://github.com/nomic-ai/gpt4all/tree/f4a1f7340c76e7d925035fda89315aaeb69f09f0

After that they changed C-API. You would need to rewrite C# bindings for that: https://github.com/nomic-ai/gpt4all/blob/main/gpt4all-backend/llmodel_c.h

AntoineCharton commented 1 year ago

To make sure I understand. Is LlmNative.cs a gpt4all script or your own binding? Looks like they did a lot of refactoring around that. The equivalent now in the samples seem to be Nativemethods.cs

Macoron commented 1 year ago

To make sure I understand. Is LlmNative.cs a gpt4all script or your own binding? Looks like they did a lot of refactoring around that. The equivalent now in the samples seem to be Nativemethods.cs

All .cs files are written by me for Unity. LlmNative contains bindings to C functions from llmmodel_c. As you said there a lot of changes, so current bindings won't work.

I believe gpt4all has its own C# bindings, however I didn't use them yet.

AntoineCharton commented 1 year ago

Yeah the bindings projects don't seems to work. Looks like someone did a whole new build pipeline for them but it's not documented anywhere. https://app.circleci.com/pipelines/github/mvenditto/gpt4all Gonna ask how is it supposed to be used.

Elevons commented 1 year ago

Would love to have this also support the new Orca interface as well.

chadb101 commented 1 year ago

can we expect an update version of this at any point? tried to update myself w/ current gpt4 to bit quant models working but cant do it

AntoineCharton commented 1 year ago

It's actually a big job. Not just about recompiling everything. I have only a few hours a week to spend on this if you need it right now you will have to do it yourself. Maybe ask the gpt 4 all team about your issue?

Elevons commented 1 year ago

I am offering a $400 bounty to anyone who can update this plugin to support the models that Gpt4all supports.

dreemur99 commented 1 year ago

It's actually a big job. Not just about recompiling everything. I have only a few hours a week to spend on this if you need it right now you will have to do it yourself. Maybe ask the gpt 4 all team about your issue?

I'm pretty sure they were talking to Macoron.

In either case, would love to see this updated to modern GPT4All as well. Growing demand for this integration. Dude should start a Patreon.

baroquedub commented 1 year ago

Adding my support for the update request. And all in on the idea of a patreon too.

I don't know much about these things but I find Orca and Falcon much quicker and less resource intensive (running in the windows installer UI) whereas this project hits my CPU hard and is slow to respond. What a great project though. Thank you @Macoron

Bl4ckh34d commented 9 months ago

Same here, demand is definitely there. Good opportunity to earn some good bucks.