Question with keypoint detection code

[puhuk]

From below code, it outputs value of keypoints by (heatmap * grid).sum(dim=(2, 3)) Then I can get tensor with shape of (10, 2) (Here, 10 is the number of keypoints)

grid = make_coordinate_grid(shape[2:], heatmap.type()).unsqueeze_(0).unsqueeze_(0) value = (heatmap * grid).sum(dim=(2, 3))

1) It seems that is p_k for each points. But I do not understand how that means the keypoints.

2) And I also do not understand how jacobian matrix could be calculated from below code

jacobian = heatmap * jacobian_map jacobian = jacobian.view(final_shape[0], final_shape[1], 4, -1) jacobian = jacobian.sum(dim=-1) jacobian = jacobian.view(jacobian.shape[0], jacobian.shape[1], 2, 2)

Could you explain with this.

[AliaksandrSiarohin]

1. It is just a mean of the heatmap, if you imagine that you have a circle as a heatmap center of the circle will be a heatmap. This operation called softargmax and this is pretty standard operation for keypoint detection.
2. We predict a jacobian for each pixel, since we care only about jacobian around keypoint, we weight the jacobian by a heatmap.

[puhuk]

Thanks! May I ask one more about calculating jacobian.

How we can get the jacobian by multipling heatmap and jacobian_map. I'm curious because heatmap is predicted area after Hourglass and conv2D layer, jacobian_map is feature_map after convolution with bias [1,0,0,1].

Sorry for dumb question, but could you explain how this output the jacobian.

[elory-y]

Do you understand the code of the Jacobian matrix calculation?