Question on differences with artidoro/qlora

[gptzerozero]

I notice that there are some differences compared to the artido/qlora repo. Why were the following code left out in this repo?

def find_all_linear_names(args, model):
    cls = bnb.nn.Linear4bit if args.bits == 4 else (bnb.nn.Linear8bitLt if args.bits == 8 else torch.nn.Linear)
    lora_module_names = set()
    for name, module in model.named_modules():
        if isinstance(module, cls):
            names = name.split('.')
            lora_module_names.add(names[0] if len(names) == 1 else names[-1])

    if 'lm_head' in lora_module_names: # needed for 16-bit
        lora_module_names.remove('lm_head')
    return list(lora_module_names)

https://github.com/artidoro/qlora/blob/3da535abdfaa29a2d0757eab0971664ed2cd97e8/qlora.py#L221-L232

and

    for name, module in model.named_modules():
        if isinstance(module, LoraLayer):
            if args.bf16:
                module = module.to(torch.bfloat16)
        if 'norm' in name:
            module = module.to(torch.float32)
        if 'lm_head' in name or 'embed_tokens' in name:
            if hasattr(module, 'weight'):
                if args.bf16 and module.weight.dtype == torch.float32:
                    module = module.to(torch.bfloat16)
    return model

https://github.com/artidoro/qlora/blob/3da535abdfaa29a2d0757eab0971664ed2cd97e8/qlora.py#L334-L344

Instead, you simply set the lora target modules to be ["q_proj", "v_proj"].

Thank you for any clarifications

[vihangd]

@gptzerozero Thanks for asking. I was planning to clarify some of the differences in the readme.md but haven't gotten around to yet. This repository is inspired by alpaca-lora and mimics some of its features or limitations. One of the future goals is to implement linear modules if there is enough demand for it. However, this repository is already a good resource for anyone who wants to experiment with different LLMs and get the most out of the consumer hardware.