Kandinsky 2.2: LoRA method - LoRAAttnAddedKVProcessor Decaprated

[erfansoheil]

Describe the bug

I have install diffusers from source and I am trying to train kandisnky 2.2 decoder with LoRa method but I get this error.

Traceback (most recent call last): File "/home/erfan/TORUS/sd_finetune/kan2/train_text_to_image_lora_decoder.py", line 812, in main() File "/home/erfan/TORUS/sd_finetune/kan2/train_text_to_image_lora_decoder.py", line 638, in main model_pred = unet(noisy_latents, timesteps, None, added_cond_kwargs=added_cond_kwargs).sample[:, :4] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/erfan/miniconda3/envs/kan2_train/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/erfan/miniconda3/envs/kan2_train/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/erfan/miniconda3/envs/kan2_train/lib/python3.12/site-packages/diffusers/models/unets/unet_2d_condition.py", line 1216, in forward sample, res_samples = downsample_block( ^^^^^^^^^^^^^^^^^ File "/home/erfan/miniconda3/envs/kan2_train/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/erfan/miniconda3/envs/kan2_train/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/erfan/miniconda3/envs/kan2_train/lib/python3.12/site-packages/diffusers/models/unets/unet_2d_blocks.py", line 2026, in forward hidden_states = attn( ^^^^^ File "/home/erfan/miniconda3/envs/kan2_train/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl return self._call_impl(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/erfan/miniconda3/envs/kan2_train/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl return forward_call(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/erfan/miniconda3/envs/kan2_train/lib/python3.12/site-packages/diffusers/models/attention_processor.py", line 519, in forward return self.processor( ^^^^^^^^^^^^^^^ File "/home/erfan/miniconda3/envs/kan2_train/lib/python3.12/site-packages/diffusers/models/attention_processor.py", line 2077, in call deprecate( File "/home/erfan/miniconda3/envs/kan2_train/lib/python3.12/site-packages/diffusers/utils/deprecation_utils.py", line 18, in deprecate raise ValueError( ValueError: The deprecation tuple ('LoRAAttnAddedKVProcessor', '0.26.0', 'Make sure use AttnAddedKVProcessor instead by settingLoRA layers to self.{to_q,to_k,to_v,add_k_proj,add_v_proj,to_out[0]}.lora_layer respectively. This will be done automatically when using LoraLoaderMixin.load_lora_weights') should be removed since diffusers' version 0.28.0.dev0 is >= 0.26.0 Steps: 0%| | 0/43100 [00:00<?, ?it/s] Traceback (most recent call last): File "/home/erfan/miniconda3/envs/kan2_train/bin/accelerate", line 8, in sys.exit(main()) ^^^^^^ File "/home/erfan/miniconda3/envs/kan2_train/lib/python3.12/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main args.func(args) File "/home/erfan/miniconda3/envs/kan2_train/lib/python3.12/site-packages/accelerate/commands/launch.py", line 1023, in launch_command simple_launcher(args) File "/home/erfan/miniconda3/envs/kan2_train/lib/python3.12/site-packages/accelerate/commands/launch.py", line 643, in simple_launcher raise subprocess.CalledProcessError(returncode=process.returncode, cmd=cmd) subprocess.CalledProcessError: Command '['/conda_environment/for/training, 'train_text_to_image_lora_decoder.py', '--pretrained_decoder_model_name_or_path=/path/to_the_local/kandisty2.2_decoder_model', '--pretrained_prior_model_name_or_path=/path/to_the_local/kandisty2.2_prior_model, '--dataset_name=/path/to/my/local/dataset', '--resolution=768', '--train_batch_size=1', '--num_train_epochs=100', '--checkpointing_steps=1000', '--learning_rate=1e-04', '--lr_scheduler=constant', '--lr_warmup_steps=0', '--seed=42', '--rank=4', '--gradient_checkpointing', '--output_dir=out_put_directory']' returned non-zero exit status 1. installed

Reproduction

export MODEL_NAME_DECODER="/path/to/kandisty2.2_decoder_model" export MODEL_NAME_PRIROR="/path/to/kandisty2.2_prior_model/"\ export DATASET_NAME="/path/to/my/local/dataset"\ export OUTPUT_DIR="output_directory" \

accelerate launch --mixed_precision="fp16" train_text_to_image_lora_decoder.py \ --pretrained_decoder_model_name_or_path=$MODEL_NAME_DECODER \ --pretrained_prior_model_name_or_path=$MODEL_NAME_PRIROR --dataset_name=$DATASET_NAME --resolution=768 \ --train_batch_size=1 --num_train_epochs=100 --checkpointing_steps=1000 --learning_rate=1e-04 --lr_scheduler="constant" --lr_warmup_steps=0 --seed=42 --rank=4 --gradient_checkpointing --output_dir=$OUTPUT_DIR

Logs

No response

System Info

Diffusers Version=0.28.dev Torch version= 2.2.1+cu118 Python version= 3.12.2

Who can help?

@sayakpaul

[sayakpaul]

You should try adapting the LoRA trainers of SD and SDXL and make adjustments to the Kandinsky scripts accordingly. If that's not an option, I think it's best to downgrade diffusers and make it work.

Cc: @yiyixuxu

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[yiyixuxu]

@erfansoheil feel free to send a PR if you managed to adapt the Kandinsky script:)

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.