search

huggingface / diffusers

🤗 Diffusers: State-of-the-art diffusion models for image and audio generation in PyTorch and FLAX.
https://huggingface.co/docs/diffusers
Apache License 2.0
23.98k stars 4.93k forks source link

[Tracker] use micro-conditioning for the SDXL trainers #6586

Open sayakpaul opened 5 months ago

sayakpaul commented 5 months ago

SDXL makes use of micro-conditioning, and it does have quite a bit of an effect on the end results. For more details, refer to the paper here.

Currently, not all of our SDXL trainers don't make use of micro-conditioning. So, it'd be nice to have micro-conditioning support as in https://github.com/huggingface/diffusers/blob/main/examples/text_to_image/train_text_to_image_lora_sdxl.py.

Below is a list of the training scripts where we'd like to have this change incorporated:

Feel free to open PRs targeting only ONE example at a time and tag me. Please also share an example training command while submitting the PRs. The command doesn't have to run the training for a large number of steps. Anything in the range of [4, 10] should suffice.

sayakpaul commented 5 months ago

@jiqing-feng, would you like to take Textual inversion SDXL?

@linoytsaban would you like to take Advanced SDXL trainer?

charchit7 commented 5 months ago

Hey @sayakpaul what's the min requirement for GPU while testing for the training?

sayakpaul commented 5 months ago

DreamBooth SDXL LoRA should run in Colab. Refer to the associated readme for more details.

linoytsaban commented 5 months ago

@linoytsaban would you like to take Advanced SDXL trainer?

sure!

jiqing-feng commented 5 months ago

@jiqing-feng, would you like to take Textual inversion SDXL?

@linoytsaban would you like to take Advanced SDXL trainer?

Sorry for the delay, I have been quite busy recently; I will do it once I have time.

ChengsongLu commented 5 months ago

DreamBooth SDXL LoRA should run in Colab. Refer to the associated readme for more details.

Can it run locally?

sayakpaul commented 5 months ago

You need to repurpose the code accordingly for that.

cosmo3769 commented 4 months ago

Hello @sayakpaul, I would love to work on ControlNet SDXL. Thank you.

sayakpaul commented 4 months ago

Please go ahead.

jiqing-feng commented 4 months ago

Hi @sayakpaul . I think the micro conditioning is already in the example, see conditioning_image_size and conditioning_crop_size, and they are all in the add_time_ids.

BTW, I found a minor error about the resize function, and I already fixed it in #7095

sayakpaul commented 4 months ago

You are right. Thanks for the quick fix.

github-actions[bot] commented 3 months ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.