search

johndpope / MegaPortrait-hack

Using Claude Opus to reverse engineer code from MegaPortraits: One-shot Megapixel Neural Head Avatars
https://arxiv.org/abs/2207.07621
67 stars 7 forks source link

Question on your warping implementation #54

Open yataoz opened 1 month ago

yataoz commented 1 month ago

  1. In apply_warping_field function, you add grid to warp_field. However, looks like your warp_field is obtained from WarpGenerator, which already generates warped coordinates instead of offsets. Is this a bug or did I miss something?

  2. What's your definition of predicted head pose in your implementation? Is it from canonical (world) space to camera space or from the reverse? Normally head pose is represented as RT matrix from canonical (world) space to camera space. When doing src -> canonical with backward warping, we should use RT instead of the inversed matrix, right?

  3. It's unclear in the original paper whether they apply the emotion warp filed first or the RT first. From my understanding, the emotion warp field should be in canonical space, so we should only apply emotion deformation in canonical space. This means it should be applied after RT when doing src --> canonical and before RT when doing canonical --> driving. But in your implementation, you always applied after RT regardless of src-->canonical or canonical --> driving.

johndpope commented 1 month ago

thanks for bringing this to my attention - I'm investigating.

johndpope commented 1 month ago

I create a PR - I'm working on something else - https://github.com/johndpope/IMF it's latest paper from Microsoft from a few weeks back - it's supposed to supercede denseflow / warp fields.

The EmoPortraits code was supposed to drop last month... it should clear up questions on warping. @JZArray has been outspoken about warp code - but i think the PR may hopefully resolve things.

johndpope commented 1 week ago

new code from one of the original authors just dropped https://github.com/neeek2303/EMOPortraits/blob/main/networks/volumetric_avatar/warp_generator_resnet.py#L11