search

tianweiy / DMD2

Other
333 stars 21 forks source link

Possibility for 30fps image generation? #6

Open beyarkay opened 1 month ago

beyarkay commented 1 month ago

Hey! Love your work, I was busy working with OpenDMD so I'm happy to see DMD2 has arrived (:

Do you have any insight for me about how many milliseconds each image takes to generate? I'm desperately trying to find/build a model that can get to 30fps (even if it's terrible resolution, 256x256 would be fine for me). I know the original DMD reported around 0.09s per image, but I couldn't see anything in "Improved Distribution Matching Distillation for Fast Image Synthesis" about how fast the image synthesis actually was. Any help you can give would be amazing!

tianweiy commented 1 month ago

I think we need to define what is the setting. Two things might be important

  1. is it batch-wise inference or per-image inference?
  2. what gpu are we using?

If we are using A100 and in the batch setting (with a batch size of 16) plus the small van decoder, the current one-step DMD2 SDXL model can generate images at 31 FPS with 704x704

FreedomGu commented 1 month ago

Hey Tianwei, I am wondering is the A100 you mentioned 80GB or 40 GB one? Thank you!

tianweiy commented 1 month ago

it is 80gb. Though the speed should be similar. Adding that we need the small vae decoder (the "Tiny VAE for faster decoding" option in the demo) to get this speed.

beyarkay commented 1 month ago

is it batch-wise inference or per-image inference?

per-image inference

what gpu are we using?

I'm developing on an M1 Mac, but am hoping to get something working on consumer GPUs. My current prototype uses SDXL Turbo and TinyVAE, I'm getting one frame every 0.6s (1.66fps)

In your opinion, do you think a lower resolution model would have a chance at 30fps (or something greater than 10fps?). I'd be happy to train my own model or take basically whatever tradeoffs, but 30fps is the goal.

If you've got any ideas about how I could increase the speed of the network that would be great <3 What you've made here is really inspiring.

voodoohop commented 1 month ago

@beyarkay if you're going for high fps I'd still stick with SD1.5. With the right fine-tune you can get good quality much faster than with SDXL at the moment. People have been getting stuff like 100fps with StreamDiffusion and SD1.5

beyarkay commented 1 month ago

Oh thanks! I totally overlooked SD1.5, I'll definitely check it out (: