Open balaji-skoruz opened 1 year ago
following
Use this
from monkeypatch.peft_tuners_lora_monkey_patch import replace_peft_model_with_gptq_lora_model
replace_peft_model_with_gptq_lora_model()
from peft import PeftModel
from monkeypatch.peft_tuners_lora_monkey_patch import Linear4bitLt
model = PeftModel.from_pretrained(model, lora_path, device_map={'': 'cpu'}, torch_dtype=torch.float32, is_trainable=True)
It's inside load_llama_model_4bit_low_ram_and_offload function
@johnsmith0031 what should be at the place of model, lora_path?
Yes, lora_path should be point to the path of finetuned lora model.
Got adapter_mode.bin,adapter_config.json after finetuning the vicuna 4bit 128g model , and checkpoints folders.![Screenshot 2023-06-02 172023](https://github.com/johnsmith0031/alpaca_lora_4bit/assets/68884041/e42c2e6b-29e0-48aa-abad-c73436bd09ec)
how to use this folders or files to infer the model?