search

PromtEngineer / localGPT

Chat with your documents on your local device using GPT models. No data leaves your device and 100% private.
Apache License 2.0
19.89k stars 2.22k forks source link

Option for usage of other AI models (or maybe just 13-b) #63

Open lokot0k opened 1 year ago

lokot0k commented 1 year ago

I thought that it would be cool to use last version of vicuna (vicuna-13b) instead of 7b, as it can be more efficient. Then I thought that it'll be useful for some people to use other models too. Therefore, I suggest adding functionality to support several models (or at least vicuna-13b🙃) in the future and to select them via arguments

I checked Vicuna docs and seems like 13-b version is configurable the same way as 7-b. So, allegedly, it can be easily implemented. I can implement and contribute this, but I need to know your opinion first!

PromtEngineer commented 1 year ago

Yes, that would be awesome. I think if we can implement it as an option list of all compatible models (just like Oobabooga text generation webtui) that would be great. The user can select which model to use when they run the run_localGPT file. I will appreciate it if you can help implement it.

isaiahsherfick commented 1 year ago

Consider combining this issue with #71

lokot0k commented 1 year ago

Okay, I'll dive into it soon

ashokrs commented 1 year ago

@lokot0k Are you able to succeed in running the Vicuna 13-b model in localGPT? what is the machine spec? what is the memory size of the GPU needed?

MocMilo commented 1 year ago

I managed to run Vicuna 13B on laptop CPU 12700H 64GB RAM, RTX 3070ti 8 GB, response is in about 3.5 minutes. I used: rnosov/Wizard-Vicuna-13B-Uncensored-HF-sharded (note this is shraded version of model, easier to download)