search

huggingface / diffusers

🤗 Diffusers: State-of-the-art diffusion models for image and audio generation in PyTorch and FLAX.
https://huggingface.co/docs/diffusers
Apache License 2.0
25.42k stars 5.27k forks source link

Add Insta-Flow Training support #5256

Closed isamu-isozaki closed 10 months ago

isamu-isozaki commented 1 year ago

Is your feature request related to a problem? Please describe. For hacktoberfest, I wanted to add in the Rectified Flow training objective which is what the instaflow paper, which can generate images in one forward pass, used to finetune stable diffusion.

Describe the solution you'd like I'll make a small pr and argument to examples/text_to_image based on code from here to examples/text_to_image loss. I'm thinking of testing it out with mnist to see if it works.

Additional context The links to the papers: instaflow rectified flow

Pevernow commented 1 year ago

+1

isamu-isozaki commented 1 year ago

I made a blog here for my notes on how the rectified flow works and what needs to be done for this PR. I was wrong in that this will probably need to be a community example in itself so might not finish within October

isamu-isozaki commented 1 year ago

The preliminary TODO list I have is

isamu-isozaki commented 1 year ago

@patrickvonplaten Hi! Do you think this should be 3 separate PRs?

DN6 commented 1 year ago

@isamu-isozaki Go for it 👍🏽 If you're adding to the examples folder I thinking you can make it a single PR.

isamu-isozaki commented 11 months ago

I finished making the dataset part. I am currently doing reflow and for log_validation, I think I will need a new scheduler for this which just adds the output of the unet /total timesteps to the latent.

github-actions[bot] commented 11 months ago

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

spacewalkingninja commented 10 months ago

I finished making the dataset part. I am currently doing reflow and for log_validation, I think I will need a new scheduler for this which just adds the output of the unet /total timesteps to the latent.

hey, hows it going, got a model now?

isamu-isozaki commented 10 months ago

@spacewalkingninja Ah I made a training colab for a lora version and training rectified flows. I did notice the loss goes down but haven't tested for that much longer training yet. Honestly, if anyone can run this for a while that'll be great. I will be running on my end too

haofanwang commented 5 months ago

It's really helpful, thanks.