Attempting to use 13B in the simple tuner -

[Atlas3DSS]

updated the main.py with decapoda-research/llama-13b-hf in all the spots that had 7B It downloaded the sharded parts all right but now im getting this config issue tho. Any advice would be appreciated.

File "/home/orwell/miniconda3/envs/llama-finetuner/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread return await future File "/home/orwell/miniconda3/envs/llama-finetuner/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run result = context.run(func, args) File "/home/orwell/miniconda3/envs/llama-finetuner/lib/python3.10/site-packages/gradio/helpers.py", line 587, in tracked_fn response = fn(args) File "/home/orwell/simple-llama-finetuner/main.py", line 82, in generate_text load_peft_model(peft_model) File "/home/orwell/simple-llama-finetuner/main.py", line 35, in load_peft_model model = peft.PeftModel.from_pretrained( File "/home/orwell/miniconda3/envs/llama-finetuner/lib/python3.10/site-packages/peft/peft_model.py", line 135, in from_pretrained config = PEFT_TYPE_TO_CONFIG_MAPPING[PeftConfig.from_pretrained(model_id).peft_type].from_pretrained(model_id) File "/home/orwell/miniconda3/envs/llama-finetuner/lib/python3.10/site-packages/peft/utils/config.py", line 101, in from_pretrained raise ValueError(f"Can't find config.json at '{pretrained_model_name_or_path}'") ValueError: Can't find config.json at ''

The config file appears in the cache the same as it does for 7B - im assuming im missing something just not sure what.

Thank you again

[mouchourider]

You need to change the requirement.txt file to this:

datasets loralib sentencepiece git+https://github.com/zphang/transformers@c3dc391 accelerate bitsandbytes git+https://github.com/huggingface/peft.git gradio

And you need to change those three functions as follow in the main.py file:

def load_base_model(): global model print('Loading base model...') model = transformers.LLaMAForCausalLM.from_pretrained( 'decapoda-research/llama-13b-hf', load_in_8bit=True, torch_dtype=torch.float16, device_map={'':'cuda'} )

def load_tokenizer(): global tokenizer print('Loading tokenizer...') tokenizer = transformers.LLaMATokenizer.from_pretrained( 'decapoda-research/llama-13b-hf', )

def load_peft_model(model_name): global model print('Loading peft model ' + model_name + '...') model = peft.PeftModel.from_pretrained( model, model_name, torch_dtype=torch.float16, device_map={'':0} )

It should work.

[Atlas3DSS]

We are working - altho i must say as it turns out it was a UI issue i think In the interference tab The lora model must have NOTHING in the field, not "none", which is what i had on mine, else it cannot find the config file. Turns out that was the issue. I clicked hte lil X and then it started working. No idea why on that one. I think it might be because up to this point i only have 7B LoRa so when that lil thing says none it is looking in 7B folder not 13B?

In either case it is now working - i thank you for your help.