A question about the computation of jacobians.

[PangzeCheung]

https://github.com/AliaksandrSiarohin/first-order-model/blob/78079d14396052dea55a1ee6a4432fb13da61c53/modules/keypoint_detector.py#L69

I find that the Heatmap is used in predicting Jacobian matrices. Why not directly using Jacobian_map to get Jacobian matrix by average operation? Thank you very much~

[AliaksandrSiarohin]

I did not try and it is hard to say what will work better. But my intuition was that with this we focus our prediction near the keypoint. Indeed why would you want to consider parts of the image that lie far away from the keypoint if the concept of these jacobians is prediction near the keypoint. But you can test you version maybe it will also work.

[PangzeCheung]

Thank you very much for your answer. I have another question about the testing process. I notice that in your paper, you use the difference between the D1 and S1 to eliminate body shape differences between different people. As you mentioned, this only applies when S1 and D1 have similar postures. I'd like to ask what results will be obtained if we test according to formula 4. Thak you~

[AliaksandrSiarohin]

If you remove flag relative from you will see.

[PangzeCheung]

@AliaksandrSiarohin What I want to ask is that if S1 and D1 do not have the same posture, we can only test by formula 4. Have you tried this and what kind of results will appear. Thank you~

[AliaksandrSiarohin]

What do you mean, what results? In the notebook it is called absolute motion, you can check it out.

[PangzeCheung]

@AliaksandrSiarohin Thanks for your rapid reply, I've solved this puzzle from your paper. I have one last question. I want to ask how this formula is obtained. I think the T(St<-R)T(R<-S1) is not equal to the T(Dt<-R)T(R<-D1) same as the jacobians because the positions and shapes of the characters in D and S are different. Thanks~

[AliaksandrSiarohin]

This a derivative of the first equation in this image, why are you asking about T(St<-R)*T(R<-S1)?

[PangzeCheung]

Thank you for your advice，I wrongly calculated the derivative of the second equation in this image.

[RAJA-PARIKSHAT]

https://github.com/AliaksandrSiarohin/first-order-model/blob/78079d14396052dea55a1ee6a4432fb13da61c53/modules/keypoint_detector.py#L69

I find that the Heatmap is used in predicting Jacobian matrices. Why not directly using Jacobian_map to get Jacobian matrix by average operation? Thank you very much~

@PangzeCheung I am trying to understand what is meant by jacobian in this implementation. The jacobian is implemented using a CNN layer. Basically, I couldn't,t intuitively understand why we are computing it and calling it jacobian, although mathematically jacobian represent different things.

[PangzeCheung]

https://github.com/AliaksandrSiarohin/first-order-model/blob/78079d14396052dea55a1ee6a4432fb13da61c53/modules/keypoint_detector.py#L69

I find that the Heatmap is used in predicting Jacobian matrices. Why not directly using Jacobian_map to get Jacobian matrix by average operation? Thank you very much~

@PangzeCheung I am trying to understand what is meant by jacobian in this implementation. The jacobian is implemented using a CNN layer. Basically, I couldn't,t intuitively understand why we are computing it and calling it jacobian, although mathematically jacobian represent different things.

I think it is maybe hard to explain this mathematically. The author just wants to use CNN as a black box to simulate the jacobian which cannot be obtained by ordinary means.

[RAJA-PARIKSHAT]

https://github.com/AliaksandrSiarohin/first-order-model/blob/78079d14396052dea55a1ee6a4432fb13da61c53/modules/keypoint_detector.py#L69

I find that the Heatmap is used in predicting Jacobian matrices. Why not directly using Jacobian_map to get Jacobian matrix by average operation? Thank you very much~

@PangzeCheung I am trying to understand what is meant by jacobian in this implementation. The jacobian is implemented using a CNN layer. Basically, I couldn't,t intuitively understand why we are computing it and calling it jacobian, although mathematically jacobian represent different things.

I think it is maybe hard to explain this mathematically. The author just wants to use CNN as a black box to simulate the jacobian which cannot be obtained by ordinary means.

@PangzeCheung So, I got this that it is basically a jacobian approximation but in the paper, this jacobian is basically d/dp[TS<-R(p)] at pk. Now, do you have any idea what is this T transformation, and what is p as highlighted in the paper.

[PangzeCheung]

https://github.com/AliaksandrSiarohin/first-order-model/blob/78079d14396052dea55a1ee6a4432fb13da61c53/modules/keypoint_detector.py#L69

I find that the Heatmap is used in predicting Jacobian matrices. Why not directly using Jacobian_map to get Jacobian matrix by average operation? Thank you very much~

@PangzeCheung I am trying to understand what is meant by jacobian in this implementation. The jacobian is implemented using a CNN layer. Basically, I couldn't,t intuitively understand why we are computing it and calling it jacobian, although mathematically jacobian represent different things.

I think it is maybe hard to explain this mathematically. The author just wants to use CNN as a black box to simulate the jacobian which cannot be obtained by ordinary means.

@PangzeCheung So, I got this that it is basically a jacobian approximation but in the paper, this jacobian is basically d/dp[TS<-R(p)] at pk. Now, do you have any idea what is this T transformation, and what is p as highlighted in the paper.

I think T is a function similar to optical flow, which can transform the keypoints between source frame and dirve frame. pk is the k-th kepoint of the reference frame.