Keep getting "has no attribute '_get_submodules'" trying to apply model

[practical-dreamer]

System Info

peft version=0.6.0.dev0 accelerate version=0.22.0 transformers version=4.32.0 python version=3.8.10

platform:

        .-/+oossssoo+/-.               root@StagingVM 
    `:+ssssssssssssssssss+:`           -------------- 
  -+ssssssssssssssssssyyssss+-         OS: Ubuntu 20.04.6 LTS x86_64 
.ossssssssssssssssssdMMMNysssso.       Host: Virtual Machine Hyper-V UEFI Release v4.1 

/ssssssssssshdmmNNmmyNMMMMhssssss/ Kernel: 5.15.0-1042-azure +ssssssssshmydMMMMMMMNddddyssssssss+ Uptime: 33 mins /sssssssshNMMMyhhyyyyhmNMMMNhssssssss/ Packages: 699 (dpkg), 4 (snap) .ssssssssdMMMNhsssssssssshNMMMdssssssss. Shell: bash 5.0.17 +sssshhhyNMMNyssssssssssssyNMMMysssssss+ Resolution: 1024x768 ossyNMMMNyMMhsssssssssssssshmmmhssssssso CPU: AMD EPYC 7763 (8) @ 2.445GHz ossyNMMMNyMMhsssssssssssssshmmmhssssssso Memory: 643MiB / 257691MiB +sssshhhyNMMNyssssssssssssyNMMMysssssss+ .ssssssssdMMMNhsssssssssshNMMMdssssssss.
/sssssssshNMMMyhhyyyyhdNMMMNhssssssss/
+sssssssssdmydMMMMMMMMddddyssssssss+ /ssssssssssshdmNNNNmyNMMMMhssssss/ .ossssssssssssssssssdMMMNysssso. -+sssssssssssssssssyyyssss+- :+ssssssssssssssssss+: .-/+oossssoo+/-.

Who can help?

@pacman100 @younesbelkada

Information

• [ ] The official example scripts
• [X] My own modified scripts

Tasks

• [ ] An officially supported task in the examples folder
• [X] My own task or dataset (give details below)

Reproduction

here's the script

from dataclasses import dataclass, field
from typing import Optional

import peft
import torch
from peft import PeftConfig, PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, HfArgumentParser

base_model_name_or_path = "TheBloke/Llama-2-7B-fp16"
adapter_path = "Xilabs/llama-2-7B-Guanaco-QLoRA"
merged_model_path = "/workspace/models/TheBloke_Llama-2-7B-fp16_Guan"

peft_config = PeftConfig.from_pretrained(adapter_path)
model = AutoModelForCausalLM.from_pretrained(
    peft_config.base_model_name_or_path,
    return_dict=True,
    torch_dtype=torch.float16,
    cache_dir='/workspace/cache'
)
tokenizer = AutoTokenizer.from_pretrained(peft_config.base_model_name_or_path)

# Load the LoRA model
model = PeftModel.from_pretrained(model, adapter_path)
model.eval()

key_list = [key for key, _ in model.base_model.model.named_modules() if "lora" not in key]
for key in key_list:
    parent, target, target_name = model.base_model._get_submodules(key)
    if isinstance(target, peft.tuners.lora.Linear):
        bias = target.bias is not None
        new_module = torch.nn.Linear(target.in_features, target.out_features, bias=bias)
        model.base_model._replace_module(parent, target_name, new_module, target)

model = model.base_model.model

model.save_pretrained(merged_model_path)

Expected behavior

Model Downloads and loads into RAM (observed through htop)

Then a few minutes later met with

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/peft/tuners/lora.py", line 492, in __getattr__
    return super().__getattr__(name)  # defer to nn.Module's logic
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1614, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LoraModel' object has no attribute '_get_submodules'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "merge-adapter2.py", line 28, in <module>
    parent, target, target_name = model.base_model._get_submodules(key)
  File "/usr/local/lib/python3.8/dist-packages/peft/tuners/lora.py", line 494, in __getattr__
    return getattr(self.model, name)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1614, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LlamaForCausalLM' object has no attribute '_get_submodules'

[BenjaminBossan]

Could you please tell us what you're trying to achieve with your code (after model.eval())? Also, please always add the full stacktrace, not just the last bit. Thanks.

[practical-dreamer]

Could you please tell us what you're trying to achieve with your code (after model.eval())? Also, please always add the full stacktrace, not just the last bit. Thanks.

That was the full stacktrace. I'm just trying to merge an lora with a model and save it. as detailed in these lines

base_model_name_or_path = "TheBloke/Llama-2-7B-fp16"
adapter_path = "Xilabs/llama-2-7B-Guanaco-QLoRA"
merged_model_path = "/workspace/models/TheBloke_Llama-2-7B-fp16_Guan"

Here's the entire log from bash to bash:

root@StagingVM:/workspace/scripts# python3 /workspace/scripts/merge-adapter2.py
Downloading (…)/adapter_config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 525/525 [00:00<00:00, 381kB/s]
Downloading (…)lve/main/config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 609/609 [00:00<00:00, 550kB/s]
Downloading (…)fetensors.index.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 26.8k/26.8k [00:00<00:00, 49.0MB/s]
Downloading (…)of-00002.safetensors: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 9.98G/9.98G [00:15<00:00, 639MB/s]
Downloading (…)of-00002.safetensors: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3.50G/3.50G [00:06<00:00, 578MB/s]
Downloading shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:21<00:00, 10.91s/it]
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:01<00:00,  1.76it/s]
Downloading (…)neration_config.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 188/188 [00:00<00:00, 143kB/s]
Downloading (…)okenizer_config.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 776/776 [00:00<00:00, 2.33MB/s]
Downloading tokenizer.model: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 500k/500k [00:00<00:00, 524MB/s]
Downloading (…)/main/tokenizer.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.84M/1.84M [00:00<00:00, 56.4MB/s]
Downloading (…)cial_tokens_map.json: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 414/414 [00:00<00:00, 1.06MB/s]
Downloading adapter_model.bin: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 640M/640M [00:13<00:00, 46.8MB/s]
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/peft/tuners/lora.py", line 492, in __getattr__
    return super().__getattr__(name)  # defer to nn.Module's logic
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1614, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LoraModel' object has no attribute '_get_submodules'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/workspace/scripts/merge-adapter2.py", line 28, in <module>
    parent, target, target_name = model.base_model._get_submodules(key)
  File "/usr/local/lib/python3.8/dist-packages/peft/tuners/lora.py", line 494, in __getattr__
    return getattr(self.model, name)
  File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 1614, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'LlamaForCausalLM' object has no attribute '_get_submodules'
root@StagingVM:/workspace/scripts#

[BenjaminBossan]

I'm just trying to merge an lora with a model and save it

Is there a reason why you're not using model.merge() or model.merge_and_unload() on the PEFT model? Note that merging quantized layers is, however, not supported yet but there is a PR (#851).

[practical-dreamer]

Sorry is there an example posted on how to do this? All I’m trying to do is merge a model with a Lora and save

[practical-dreamer]

I'm just trying to merge an lora with a model and save it

Is there a reason why you're not using model.merge() or model.merge_and_unload() on the PEFT model? Note that merging quantized layers is, however, not supported yet but there is a PR (#851).

AttributeError: 'LlamaForCausalLM' object has no attribute 'merge'

[BenjaminBossan]

That's probably because of what I mentioned:

merging quantized layers is, however, not supported yet but there is a PR (#851).

You could either try again based on that branch or just wait a little bit more until we merge it in PEFT, which should hopefully be very soon.

[practical-dreamer]

I’m sorry I don’t see the relevance… neither the model nor the Lora adapter are quantized.

base_model = "TheBloke/Llama-2-7B-fp16" adapter = "Xilabs/llama-2-7B-Guanaco-QLoRA"

The model contains unquantized fp16 weights… how would a PR for loading quantized layers help?

[BenjaminBossan]

Ah yes, sorry. Could you try merge_adapter, not merge?

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.