PeftModel is not an instance of PreTrainedModel. `No liger kernels will be applied.`

[ambroser53]

System Info

transformers==4.45.1 peft==0.13.0 liger-kernel==0.3.1

So isinstance(model.base_model.model, PreTrainedModel) returns true but isinstance(model, PreTrainedModel) returns false so no liger kernels are used.

Who can help?

No response

Information

• [ ] The official example scripts
• [ ] My own modified scripts

Tasks

• [ ] An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• [ ] My own task or dataset (give details below)

Reproduction

Load any model with a PeftModel wrapper via get_peft_model and try to run with the use_liger_kernel flag in the trainer.

Expected behavior

Apply liger kernels. Be as simple as add a check for peft models that then checks model.base_model.model

[LysandreJik]

cc @benjaminbossan for the PeftModel and @ArthurZucker for the liger kernel

[ambroser53]

I've narrowed the cause down to the changes made in #33502

[BenjaminBossan]

Just to confirm, a PeftModel is not an instance of PretrainedModel, so it is expected that this check would return False. As @ambroser53 correctly identified, we can check if the model is a PEFT model and if this is true, we can check:

1. isinstance(model.base_model, PreTrainedModel) -> apply liger to model.base_model (for prompt learning methods)
2. isinstance(model.base_model.model, PreTrainedModel) -> apply liger to model.base_model.model (for LoRA etc.)

I don't know what exactly liger kernels do, but from the description

We have implemented Hugging Face Compatible RMSNorm, RoPE, SwiGLU, CrossEntropy, FusedLinearCrossEntropy

they should not interfere with PEFT and should be usable in conjunction. LMK what you think.