LoRA seems not training the linear head for classification

4AI / LS-LLaMA

A Simple but Powerful SOTA NER Model | Official Code For Label Supervised LLaMA Finetuning

https://arxiv.org/abs/2310.01208

MIT License

131 stars 20 forks source link

Closed mutetea closed 9 months ago

mutetea commented 10 months ago

For llama_seq_clf，it seems you just use LlamaForSeq Classification，which use a new linear head to substitute the original head.

I don't think LoRA train this layer，why this is not an issue？

SeanLee97 commented 9 months ago

Hi @mutetea, sorry for the delayed relay.

For LlamaForSequenceClassification, we use the official implementation. The classification layer with the name classifier is automatically added to the trainable parameters in PEFT, you can check this: https://github.com/huggingface/peft/blob/main/src/peft/peft_model.py#L788-L791