search

ml-jku / EVA

One Initialization to Rule them All: Fine-tuning via Explained Variance Adaptation
MIT License
29 stars 0 forks source link

Support for Non-Language Models? #1

Open vitrun opened 2 days ago

vitrun commented 2 days ago

Hey there! Loving the research you've shared. I've been playing around with finetuning Stable Diffusion models and ran into some snags. Here's the error I got:

  File "/root/ws/peft/src/peft/tuners/lora/eva.py", line 586, in get_eva_state_dict
    eva_state_dict = _get_eva_state_dict(
  File "/root/ws/peft/src/peft/tuners/lora/eva.py", line 303, in _get_eva_state_dict
    model_inputs_for_hooks = prepare_model_inputs_fn(inputs, peft_config)
  File "/root/ws/peft/src/peft/tuners/lora/eva.py", line 218, in prepare_model_inputs_fn_language_modeling
    mask = model_input.get("attention_mask", torch.ones_like(model_input["input_ids"])).bool()
KeyError: 'input_ids'

Looks like it's set up just for language models, right? Did I miss something, or is there a way to tweak it for other types of models too? Would love to expand the use case if possible! Thanks!

sirluk commented 1 day ago

@vitrun great to hear you would like to train out EVA on Diffusion models! While the implementation in this repo does not support this, we just merged EVA into peft: https://github.com/huggingface/peft/pull/2142. So if you install peft from source you can already use it.

In the Readme you can find info about how to customize eva to your use case.