search

magic-research / magic-animate

[CVPR 2024] MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model
https://showlab.github.io/magicanimate/
BSD 3-Clause "New" or "Revised" License
10.5k stars 1.08k forks source link

non-square aspect ratios #129

Closed ddpasa closed 11 months ago

ddpasa commented 11 months ago

If I understand the animation pipeline construction correctly, the input video and picture is rescaled to a square shape before being processed by the pipeline: https://github.com/magic-research/magic-animate/blob/main/magicanimate/pipelines/animation.py#L177 This means that if you pass a picture or densepose video that is not square, then horrible aspect ratio deformations occur.

  1. Is it possible to remove this requirement? If we adjust animation.py so to handle non-square aspect rations, would the underlying model be able to handle it?

  2. If the answer to the above question is no, then what's the recommended strategy for non-square inputs? It's easy to crop photos to be square, but are these libraries that can take a non-square densepose video and make it square?

Thanks!

zcxu-eric commented 11 months ago

Hi, the answer is no, this library was only developed on square videos and all the inputs should be square.

ddpasa commented 11 months ago

Thanks @zcxu-eric for the reply. Is there a way to convert non-square densepose vides to square ones by adding more padding around the video?

zcxu-eric commented 11 months ago

Thanks @zcxu-eric for the reply. Is there a way to convert non-square densepose vides to square ones by adding more padding around the video?

Hi, of course, it is possible. We tried to pad non-square images into square using numpy 'edge' padding.