search

bublint / ue5-llama-lora

A proof-of-concept project that showcases the potential for using small, locally trainable LLMs to create next-generation documentation tools.
MIT License
499 stars 35 forks source link

Which Llama-7b model was used? #7

Open gptzerozero opened 1 year ago

gptzerozero commented 1 year ago

Which LLaMA 7b model was used?

Is it available on Hugging face?

Thank you. I am trying to reproduce your success

hypersniper05 commented 1 year ago

This project needs a bit of an update. Fine tuning in a large text file is not going to have the results you are looking for. You can now use Superbooga however use the extension https://github.com/oobabooga/text-generation-webui/tree/main/extensions%2Fsuperbooga In text generation webui. The Vicunia model you can use that is pretty good is Vicuna 1.3 7B .

gptzerozero commented 1 year ago

@hypersniper05 Thank you for introducing Superbooga! I will try it out.

For the case where the LLM 2K context window is too small to fit in the context when using superbooga, is it finetuning still the way to go?

hypersniper05 commented 1 year ago

Can you run the 13B quantized ? It's 11GB vram. Here is the link:

https://huggingface.co/TheBloke/Vicuna-13B-1-3-SuperHOT-8K-GPTQ

The superhot funetuned gives you more tokens.

I have been able to get 6k out of it with a 3080ti.

Update oobabooga, download the model via url link in models tab, set model to exllama, use 6k sequence length, 2 context, set chat instruct template to vicuna 1.1 . Select instruct on chat and setup superbooga.

If you use context 4 with 8k on my machine at least, numbers are off. The model will have a missing 0 if it quotes 00 .

hypersniper05 commented 1 year ago

I just wanted to let you know that chatgpt can help you with your prompt to get it to work better for you. Usually, I don't have to be so specific with chatgpt 3.5, but with 13B and 7B, you do. I just finished creating synthetic data for a training I want to do on a smaller 3B model (planning on running it on mobile) using the Vicuna 13B 1.3 model, and looking at my spreadsheet and comparing the same query and outputs its scary similar to turbo3.5. Turbo is slightly and I mean very slightly better, but you won't be able to tell. It's all about the prompt for these little guys.