Integrate StableDiffusion Dreambooth with ColossalAI

[0xdevalias]

WHAT

Integrate with ColossalAI for memory/speed efficiency and improvements.

WHERE

• https://github.com/hpcaitech/ColossalAI

  • Colossal-AI: A Unified Deep Learning System for Big Model Era

• https://medium.com/@yangyou_berkeley/diffusion-pretraining-and-hardware-fine-tuning-can-be-almost-7x-cheaper-85e970fe207b

HOW

• https://github.com/hpcaitech/ColossalAI/tree/main/examples/images/diffusion

SEE ALSO

• https://github.com/huggingface/diffusers/issues/1212
• https://github.com/AUTOMATIC1111/stable-diffusion-webui/issues/4606
• https://github.com/d8ahazard/sd_dreambooth_extension/issues/174
• https://github.com/hpcaitech/ColossalAI/issues/1935
• https://github.com/TheLastBen/fast-stable-diffusion/issues/493

[arpowers]

@0xdevalias i tried colossalAi and it was still pretty buggy in terms of documentation and getting it working as described

[0xdevalias]

That sucks :(

It looks like the dev team of ColossalAI is looking to work directly with huggingface diffusers to simplify integrations in this issue:

• https://github.com/huggingface/diffusers/issues/1212

This is very cool @binmakeswell ! Would be very happy to integrate diffusers in colossalAI examples!

Originally posted by @patil-suraj in https://github.com/huggingface/diffusers/issues/1212#issuecomment-1310167948

Thanks for your enthusiastic response and we look forward to working together!

How do you expect it to be carried out? Take our collaboration with Lightning for example, we communicated in slack for details, provided documentation and eventually did a joint release.

Originally posted by @binmakeswell in https://github.com/huggingface/diffusers/issues/1212#issuecomment-1311220560

And there seems to be at least some initial/cursory interest on the issue I opened on the ColossalAI repo RE: integrating with dreambooth:

• https://github.com/hpcaitech/ColossalAI/issues/1935

Hi, thanks for reaching out! I think it’s pretty awesome. Can I check it with our team and get back to you ASAP?

Originally posted by @Sze-qq in https://github.com/hpcaitech/ColossalAI/issues/1935#issuecomment-1313053594

So hopefully things rapidly move towards the 'easier to integrate'/'better documentation' direction of things!