search

PaddlePaddle / PaddleNLP

👑 Easy-to-use and powerful NLP and LLM library with 🤗 Awesome model zoo, supporting wide-range of NLP tasks from research to industrial applications, including 🗂Text Classification, 🔍 Neural Search, ❓ Question Answering, ℹ️ Information Extraction, 📄 Document Intelligence, 💌 Sentiment Analysis etc.
https://paddlenlp.readthedocs.io
Apache License 2.0
11.71k stars 2.86k forks source link

[Question]: 如何使用LoRA #8647

Open carbonatedbeverages opened 1 week ago

carbonatedbeverages commented 1 week ago

请提出你的问题

model=ppnlp.transformers.NeZhaForSequenceClassification.from_pretrained('nezha-large-wwm-chinese', num_classes=14) tokenizer = ppnlp.transformers.NeZhaTokenizer.from_pretrained('nezha-large-wwm-chinese') lora_config = LoRAConfig( r=16, target_modules=['query', 'value'], lora_alpha=32, lora_dropout=0.05, ) lora_model = LoRAModel(model, lora_config) model = lora_model 运行后显示[2024-06-21 21:58:38,253] [ INFO] - Frozen parameters: 0.00e+00 || Trainable parameters:3.25e+08 || Total parameters:3.25e+08|| Trainable:100.00% LoRA没有起作用是为什么呢？以上写法的问题在哪

DrownFish19 commented 5 days ago

target_modules: List of module names or regex expression of the module names to replace with Lora. target_modules=['query', 'value'],应该使用完整的参数名称或者正则表达式，如target_modules=['*query*', '*value*'], 可参考target_module写法