search

frank-xwang / InstanceDiffusion

[CVPR 2024] Code release for "InstanceDiffusion: Instance-level Control for Image Generation"
https://people.eecs.berkeley.edu/~xdwang/projects/InstDiff/
Apache License 2.0
515 stars 29 forks source link

How to conduct model fine-tuning training and what is the required data magnitude? #11

Closed grainw closed 8 months ago

frank-xwang commented 8 months ago

Hi, you can follow our instructions on model training to finetune the model on your datasets. If you fine-tune the pretrained InstanceDiffusion model with LORA, the data magnitude can be significant reduced. The exact data magnitude depends on your task.

grainw commented 8 months ago

Do you have any plans to open source for lora training?

frank-xwang commented 8 months ago

Hi, this repo is mainly for reproducing the results reported in our paper. Currently, we don't have plans to support additional new tasks or lora training. But you can check https://github.com/cloneofsimo/lora to learn how to add LORA to stable diffusion (the base model we used in this repo). Thanks!

66ling66 commented 8 months ago

Thanks you for your great job,Can I use your model to do simpler training on a RTX3090 24g?

frank-xwang commented 8 months ago

You may want to use flash attention (and or deepspeed) during the training time if you want to train the model using RTX 3090. Flash attention was implemented already, you can set it as True in the .yaml config file.

Hope it helps.

frank-xwang commented 8 months ago

Closing it for now, please reopen-it if you have more questions.