Once this model is trained, how should I infer ? Is there an example script?

[ScottishFold007]

Once this model is trained, how should I infer ? Is there an example script? I'm having trouble loading the model using the following method：

It seems that this model lacks many parameters, can the actual prediction be good?

[Guitaricet]

Hi! Thank you for providing more details about your question.

Assuming .merge_and_reinit() was applied before the model was saved, you can use this code:

from peft_pretraining.modeling_llama import LlamaForCausalLM
model = LlamaForCausalLM.from_pretrained("path/to/model/save/dir")

Wrapping it with ReLoRA model is only needed for training.

[datalee]

.merge_and_reinit() maybe have some wrong？ It's a big difference compared to lora.