search

huggingface / diffusers

🤗 Diffusers: State-of-the-art diffusion models for image and audio generation in PyTorch and FLAX.
https://huggingface.co/docs/diffusers
Apache License 2.0
25.14k stars 5.2k forks source link

controlNet training for dstilled models #8424

Open kmpartner opened 3 months ago

kmpartner commented 3 months ago

Is it possible to train controlNet for distilled model (Block-removed Knowledge-distilled Stable Diffusion, https://github.com/Nota-NetsPresso/BK-SDM)?

When I used scripts in example folder for distilled model, it returns error, similar to "unknown mid_block_type ...." , probably because of unet structure changed. Are there any ways to train controlNet for distilled model?

sayakpaul commented 3 months ago

Can you provide more details into how you're approaching the training? Maybe provide some code snippets, etc.?

kmpartner commented 3 months ago

Thank you for response. I just followed code in controlnet folder using distilled models (mid block deleted models, such as "nota-ai/bk-sdm-tiny")

accelerate launch train_controlnet.py \ --pretrained_model_name_or_path="nota-ai/bk-sdm-tiny" \ --dataset_name=fusing/fill50k \ --resolution=512 \ --learning_rate=1e-5 \ --validation_image "./conditioning_image_1.png" "./conditioning_image_2.png" \ --validation_prompt "red circle with blue background" "cyan circle with brown floral background" \ --train_batch_size=4

Before starting training iteration error was displayed probably because mid block is removed from unet.

" Traceback (most recent call last): File "/content/diffusers/examples/controlnet/train_controlnet.py", line 1187, in main(args) File "/content/diffusers/examples/controlnet/train_controlnet.py", line 818, in main controlnet = ControlNetModel.from_unet(unet) File "/usr/local/lib/python3.10/dist-packages/diffusers/models/controlnet.py", line 471, in from_unet controlnet = cls( File "/usr/local/lib/python3.10/dist-packages/diffusers/configuration_utils.py", line 658, in inner_init init(self, *args, **init_kwargs) File "/usr/local/lib/python3.10/dist-packages/diffusers/models/controlnet.py", line 442, in init raise ValueError(f"unknown mid_block_type : {mid_block_type}") ValueError: unknown mid_block_type : None ... "

github-actions[bot] commented 1 day ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.