search

jncraton / languagemodels

Explore large language models in 512MB of RAM
https://jncraton.github.io/languagemodels/
MIT License
1.18k stars 78 forks source link

Commercial Models? (Follow Up) #23

Closed GenerativeGuru closed 1 year ago

GenerativeGuru commented 1 year ago

Greetings,

Following up from my previous issue, I see you've added this section on commercial licenses, but I'm struggling to understand what sort of models may be returned by such a command. Do you have a list somewhere?

My real interest though is in figuring out how to make this work with the orca-3b model. I've played with it using GPT4All and llama (my goal is CPU inference on my Intel MacBook Pro) but I like your implementation the best.

I've tried looking into things, and it looks like you're using something called ctransformers2, but it's all a bit over my head still. Can you help dumb down some high level instructions I could do if I wanted to get orca-3b working with this?

jncraton commented 1 year ago

I'm struggling to understand what sort of models may be returned by such a command. Do you have a list somewhere?

This isn't well documented at the moment, but there is a list of all supported models in the source code. Right now, you'll get FLAN-T5 models if you use the regex filter from the readme.

My real interest though is in figuring out how to make this work with the orca-3b model.

I'm assuming that you are talking about orca-mini-3b. This should work with CTranslate2 relatively easily, as it is an OpenLLaMA fine-tune, but I haven't personally used the 3B LLaMA models with CTranslate2. It should work almost exactly like the other generative models. There are several examples in the CTranslate2 documentation.