how to convert joints into heatmap

[lulubbb]

Hello bro! I interrupted you suddenly because I wanted to ask how you converted 17 keypoints into a heat map (117256*256) in the dummynet project, and I can't find relevant code in this repo, could you please provide this code or another tips? thanks again!

[vobecant]

Hi, this should be the code that would generate the heatmaps from keypoints:

def get_keypoint_masks(kp, im_size, size_frac=40, joints=None, visibility=None, head_ids=[0, 1, 2, 3, 4]):
    '''
    Creates masks of keypoints. One mask per keypoint.
    :param kp: array of keypoint information
    :param im_size: size of image
    :return: array containing masks of keypoints
    '''
    x = kp[0::3] if joints is None else [x for x, y in joints]
    y = kp[1::3] if joints is None else [y for x, y in joints]
    visibility = kp[2::3] if visibility is None else [2 if v else 0 for v in visibility]
    size = int(np.ceil(im_size[0] / size_frac))
    size = max([size, 1])
    size_head = int(size * 0.75)
    size_head = max([size_head, 1])
    masks = []
    for id, v in enumerate(visibility):
        mask = np.zeros(im_size)
        mask.fill(0)
        if v > 0:
            center = (int(x[id]), int(y[id]))
            value = 1 
            if head_ids is not None and id in head_ids:
                cv2.circle(mask, center, size_head, value, -1)
            else:
                cv2.circle(mask, center, size, value, -1)
        masks.append(mask)
    return masks

[lulubbb]

@vobecant Thanks for the code! When I transfer joints into heatmap using above code, and send the heatmap to the mask_estimator, finally yeild a mask(predicted mask looks good). I meet a problem, when I input z (provide by your data), mask (generate by mask_estimator) and heatmap (generate by above code) into conditional generator, output a very poor result(first figure). But if make mask(generate by mask_estimator ) and heatmap(provide by your data) as the input of conditional generator, the output(second figure) of conditional generator looks good, z is same. I don't understrand which step result a poor output, I guess this problem maybe cause by heatmap channel? But I try to convert heatmap to skeleton and draw it, the skeleton image(third figure) is also normal. So, I really confuse about it. Can you help me, thanks again bro!

[vobecant]

Hi, if I understand it correctly, you refer to the good first image and to the bad second image. The only difference is in the way you prepare the mask? Do you use the same set of keypoints in both examples?