[Feature request]: Add SD3 Support

[Silvelter]

SD3 is out

[Lotayou]

mark

[Cauldrath]

Looks like there is example Dreambooth code in the diffusers repo: https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/train_dreambooth_sd3.py

[lzran]

Looks like there is example Dreambooth code in the diffusers repo: https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/train_dreambooth_sd3.py

but the in the diffusers repo,the example for generating image is not enough, the gen_img.py update is needed. 0.0

[zengjie617789]

I followd the official guide to train a lora with costom datasets and the results I got is bad that not show the concept at all.

[lzran]

I followd the official guide to train a lora with costom datasets and the results I got is bad that not show the concept at all.

the generation program you used for results is needed to be modifled? which file ? gen_img.py or sdxl_gen_img.py ? can I use it only change the parameters for model path ?

[ymzlygw]

mark

[zengjie617789]

@lzran I used the guide. I defined a custom datasets with image and txt pair. After trainning, just use load_lora_weights() to infer to get the results.

[Cauldrath]

I don't know how accurate this is, but some people were saying that they couldn't get SD3 to train without QK-norm: https://arxiv.org/abs/2010.04245

[leonary]

看起来在扩散器存储库中有一个 Dreambooth 示例代码：https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/train_dreambooth_sd3.py

I also used the example given by Diffuser to test training on 40GB video memory. It seems to require a very high learning rate, such as 1e-3. Even at a very high repeat, such as 300, the fitting effect is very poor, and it does not support text labels but only instance prompts, such as "a girl named Lily" or "a dog named Sks". It should be just a very basic implementation of Dreambooth, which is very different from Lora and Dreambooth in sd-scripts. At least I can advise you not to try this script again, it is almost useless unless you want to train a dog named Sks. For most of the fine-tuning authors of SD15, SDXL and other models, it will not be able to achieve any good results on your previous datasets.

[1030zero]

看起来在扩散器存储库中有一个 Dreambooth 示例代码：https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/train_dreambooth_sd3.py

我也用了Diffuser给的例子在40GB显存上测试的训练，好像需要非常高的学习率，比如1e-3，即使非常高的重复率，比如300下，符合效果也很差，而且不支持文本标签只支持实例提示，比如“一个叫Lily的女孩”或者“一只叫Sks的狗”。应该只是Dreambooth的一个非常基础的实现，和Lora以及sd-scripts中的Dreambooth课堂区别。至少我可以劝你不要再尝试这个脚本，除非你想训练一只叫Sks的狗，否则它几乎没用。对于大部分SD15、SDXL等模型的微调作者来说，它无法在你之前的数据集上取得什么好的效果。 上面的训练链接不支持文本标签，那如果是把底模更改成sd3运用这个dreambooth训练可以么？因为需要训练多个角色必须要有文本标签！

[leonary]

看起来在扩散器存储库中有一个 Dreambooth 示例代码：https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/train_dreambooth_sd3.py

我也用了Diffuser给的例子在40GB显存上测试的训练，好像需要非常高的学习率，比如1e-3，即使非常高的重复率，比如300下，符合效果也很差，而且不支持文本标签只支持实例提示，比如“一个叫Lily的女孩”或者“一只叫Sks的狗”。应该只是Dreambooth的一个非常基础的实现，和Lora以及sd-scripts中的Dreambooth课堂区别。至少我可以劝你不要再尝试这个脚本，除非你想训练一只叫Sks的狗，否则它几乎没用。对于大部分SD15、SDXL等模型的微调作者来说，它无法在你之前的数据集上取得什么好的效果。 上面的训练链接不支持文本标签，那如果是把底模更改成sd3运用这个dreambooth训练可以么？因为需要训练多个角色必须要有文本标签！ Use this project to train Lora or Finetune. I tested it and the training results fully met my expectations.It supports text tags, but there are a lot of things to configure, which requires a little learning cost. https://github.com/bghira/SimpleTuner/blob/main/documentation/quickstart/SD3.md

[1030zero]

看起来在扩散器存储库中有一个 Dreambooth 示例代码：https://github.com/huggingface/diffusers/blob/main/examples/dreambooth/train_dreambooth_sd3.py

我也用了Diffuser的例子在40GB显存上测试的训练，好像需要非常高的学习率，比如1e-3，即使非常高的重复率，比如300下，符合效果也很差，而且不支持文本标签只支持实例提示，例如“一个叫Lily的女孩”或者“一只叫Sks的狗”。应该只是Dreambooth的一个非常基础的实现，和Lora以及sd-scripts中的Dreambooth课堂区别。至少我可以劝告你不要再尝试这个脚本，除非你想训练一只叫Sks的狗，否则它几乎没用。对于大部分SD15、SDXL等模型的微调作者来说，它无法在你可用的存储空间上取得什么好效果。 上面的训练链接不支持文本标签，那如果是把底线更改成sd3运用这个dreambooth训练可以么？ 因为需要训练多个角色必须要有文本标签！ 使用此项目来训练Lora或Finetune。我测试了一下，训练结果完全符合我的预期。它支持文本标签，但是需要配置的东西比较多，需要一点学习成本。 https://github.com/bghira/SimpleTuner/blob/main/documentation/quickstart/SD3.md

当我运行这段命令时总是会有错误，但是我不知道如何解决，请问我的执行命令是正确的吗？