search

brandontrabucco / da-fusion

Effective Data Augmentation With Diffusion Models
MIT License
221 stars 18 forks source link

Where in the code is SDEdit? #18

Open jmugan opened 9 months ago

jmugan commented 9 months ago

Hi, love the project. I'm trying to run this on a custom dataset, and I can't see where the image is inserted into the diffusion process as described in equation 6 in the paper. I see the textual inversion but not that. Can someone point me to where that is? Thanks!

brandontrabucco commented 9 months ago

Hello Jonathan,

Thanks for your interest in our code! SDEdit is implemented inside the diffusers StableDiffusionImg2ImgPipeline class: https://huggingface.co/docs/diffusers/v0.26.3/en/api/pipelines/stable_diffusion/img2img#diffusers.StableDiffusionImg2ImgPipeline. The key arguments to this pipeline's call method for generation are image and strength, the latter of which controls the extent the images are modified.

This pipeline is used in the code for augmentation here: https://github.com/brandontrabucco/da-fusion/blob/dbcdc9c6aeb0a25aa1cb2bdddc7104be2ee63bf4/semantic_aug/augmentations/textual_inversion_upstream.py#L62

-Brandon

brandontrabucco commented 9 months ago

Also, when loading textual inversion embeddings, the diffusers Stable Diffusion pipelines will fail silently if a Textual Inversion embedding doesn't exist in the folder the code points to: https://github.com/brandontrabucco/da-fusion/blob/dbcdc9c6aeb0a25aa1cb2bdddc7104be2ee63bf4/semantic_aug/augmentations/textual_inversion_upstream.py#L76

Make sure to use the fine_tune_upstream.py script to train these embeddings first.

Let me know how else I can help!

-Brandon

jmugan commented 9 months ago

Thanks! Still working through it.