How to use inference.py after finetune.py?

[athenawisdoms]

I've completed my first finetune using finetune.py with params

    --lora_out_dir=./my_lora/ \
    --llama_q4_config_dir=./Neko-Institute-of-Science_LLaMA-30B-4bit-128g \
    --llama_q4_model=./Neko-Institute-of-Science_LLaMA-30B-4bit-128g/llama-30b-4bit-128.safetensors \

How should I modify inference.py to do a quick test on the newly trained lora at ./my_lora/?

I think I need to change model_path and config_path, but which directories should I point them to?

[johnsmith0031]

You can use this function and set lora_path to your model. load_llama_model_4bit_low_ram_and_offload

[athenawisdoms]

Works and thanks!