search

AliaksandrSiarohin / monkey-net

Animating Arbitrary Objects via Deep Motion Transfer
467 stars 81 forks source link

keypoint predictions #28

Closed saandeepa93 closed 11 months ago

saandeepa93 commented 3 years ago

Hello, I am currently generating keypoints on a different face dataset and they look a bit off (they do not focus on the supposedly moving parts). They also generate varying keypoints at each run. I have tried with different normalizing constants while apply softmax to the heatmaps but it doesn't seem to focus. The overall U-net architecture is pretty much the same.

Can you please provide any suggestion to improve this?

Thank you.

AliaksandrSiarohin commented 3 years ago

Use perceptual loss similar to fomm https://github.com/AliaksandrSiarohin/first-order-model/blob/78079d14396052dea55a1ee6a4432fb13da61c53/modules/model.py#L10

saandeepa93 commented 3 years ago

Thank you for your timely responses. Can you also point me to some of the related resources/papers that talks more about the dense optical flow calculations in general? I am a little confused with the usage of masks in all this.

AliaksandrSiarohin commented 3 years ago

Dense optical flow is our definition and it is only used in Monkey-Net and in FOMM Sec 3.1 (https://arxiv.org/abs/2003.00196).

JialeTao commented 3 years ago

Dense optical flow is our definition and it is only used in Monkey-Net and in FOMM Sec 3.1 (https://arxiv.org/abs/2003.00196).

Hi, it seems the Jacobian matrix of size 2*2 is also named as "affine transformation" in FOMM, and sometimes the optical flow is also explained as "affine transformation". It's a little confused for me to distinguish those and what are the association between those concepts?

AliaksandrSiarohin commented 3 years ago

Optical flow is not an affine transformation, can you indicate the line which you did not understand?

JialeTao commented 3 years ago

Optical flow is not an affine transformation, can you indicate the line which you did not understand?

Hi, I just get the concept of optical flow, it is a coordinate mapping from target frame to source frame with size HW2 ,right? And I'm now confused about the relathiship between Jacobian matrix and affine transformation. Noted that in the figure2 of FOMM, Jacobian matrix is also instructed as affine transformation.

AliaksandrSiarohin commented 3 years ago

Jacobians matrix and position of keypoints together construct a representation of affine transform. This transform represents the movement of particular part.