more tensor bugs in train_dreambooth_lora_flux.py

[neuron-party]

Describe the bug

after hardcoding a workaround regarding transformer.config.guidance_embeds, more tensor errors appear

Reproduction

--resolution=1024
--mixed_precision=bf16
--pretrained_model_name_or_path=black-forest-labels/FLUX.1-dev
--num_validation_images=8
--validation_epochs=100
--rank=16
--train_batch_size=1
--learning_rate=1e-4
--guidance_scale=3.5
--checkpointing_steps=200
--instance_prompt=xyz
--instance_data_dir=xyz
--output_dir=xyz
--logging_dir=xyz
--validation_prompt=xyz

config file (deepspeed)

compute_environment: LOCAL_MACHINE
deepspeed_config:
  gradient_accumulation_steps: 1
  gradient_clipping: 1.0
  offload_optimizer_device: none
  offload_param_device: none
  zero3_init_flag: true
  zero_stage: 3
distributed_type: DEEPSPEED
fsdp_config: {}
machine_rank: 0
main_process_ip: null
main_process_port: null
main_training_function: main
mixed_precision: fp16
num_machines: 1
num_processes: 8
use_cpu: false

Logs

instance_prompt_hidden_states, instance_pooled_prompt_embeds, instance_text_ids = compute_text_embeddings()
->
prompt_embeds, pooled_prompt_embeds, text_ids = encode_prompt()
->
pooled_prompt_embeds = _encode_prompt_with_clip()
-> 
prompt_embeds = text_encoder(text_input_ids.to(device), output_hidden_states=False)
-> 
RuntimeError: 'weight' must be 2-D

System Info

diffusers from source, transformer==4.44.1 peft==0.10.0 accelerate==0.33.0

Who can help?

@sayakpaul @linoytsaban

[linoytsaban]

Hey @neuron-party, are you using the updated version (from this https://github.com/huggingface/diffusers/pull/9139 merged PR)? can you please elaborate on:

after hardcoding a workaround regarding transformer.config.guidance_embeds

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.