JimmySuen / integral-human-pose

Integral Human Pose Regression
MIT License
471 stars 76 forks source link

Integral regression result using softmax will be affected by the location p where the heatmap value is 0. #8

Closed r1c7 closed 5 years ago

r1c7 commented 5 years ago

We only consider one keypoint. If the heatmap is [[0. , 0. , 0. , 0. , 0. , 0. , 0. ], [0. , 0. , 0. , 0. , 0. , 0. , 0. ], [0. , 0. , 0. , 0. , 0. , 0. , 0. ], [0. , 0.1 , 0. , 0. , 0. , 0. , 0. ], [0.2 , 0.3 , 0.2 , 0.2 , 0. , 0. , 0. ], [0.1 , 0.4 , 0.05, 0. , 0. , 0. , 0. ], [0.2 , 0.2 , 0.2 , 0.2 , 0. , 0. , 0. ]]

After using the softmax, the normalized heatmap become [[0.01935992, 0.01935992, 0.01935992, 0.01935992, 0.01935992, 0.01935992, 0.01935992], [0.01935992, 0.01935992, 0.01935992, 0.01935992, 0.01935992, 0.01935992, 0.01935992], [0.01935992, 0.01935992, 0.01935992, 0.01935992, 0.01935992, 0.01935992, 0.01935992], [0.01935992, 0.02139602, 0.01935992, 0.01935992, 0.01935992, 0.01935992, 0.01935992], [0.02364626, 0.02613315, 0.02364626, 0.02364626, 0.01935992, 0.01935992, 0.01935992], [0.02139602, 0.0288816 , 0.02035252, 0.01935992, 0.01935992, 0.01935992, 0.01935992], [0.02364626, 0.02364626, 0.02364626, 0.02364626, 0.01935992, 0.01935992, 0.01935992]] .

So integral regression result will be affected by the value 0.01935992 which is calculated using softmax.

lck1201 commented 5 years ago

When using integral, only after softmax, value in heatmap stands for possibility, before that, the value doesn't have any semantics. In fact, if you print the direct output of network, there are many negatives, but it doesn't matter. @crsailing

r1c7 commented 5 years ago

If we use heatmap of groundtruth to test, using the integral after softmax, we cannot get the accurate label position.

lck1201 commented 5 years ago

@crsailing The GT heatmap you assumed, in fact, is totally not the same with prediction of the network. Try to print the direct output of network, and see what the heatmap looks like. By the way, the integral is a new method, not related to onehot/gaussian heatmap, so from this perspective, you cannot assume a GT heatmap, applying integral on it, and expect getting accurate position. It doesn't make any sense.

chaneyddtt commented 4 years ago

Hi, if the output is not like a gaussian heatmap, how do you train the network with both heatmap and intergral loss, namely I1 in your paper. The output should be something like a gaussian heatmap since you supervise the network with the gaussian heatmap before the intergral operation?

lck1201 commented 4 years ago

@chaneyddtt Hi, in practice, we train the network using gaussian loss (H1) and finetune with integral loss(l1)

For integral regression methods (I1, I2, I3, and their multi-stage versions), the network is pre-trained only using heat map loss (thus their H versions) and then, only integral loss is used. We found this training strategy working slightly better than training from scratch using both losses.

chaneyddtt commented 4 years ago

Ok, thanks for clarifying.

Ericjiejie commented 4 years ago

@lck1201 Hi, is the heatmap on the right column of Figure 2 in the your paper(https://arxiv.org/pdf/1711.08229v1.pdf) ouput from network or softmax?

lck1201 commented 4 years ago

@Ericjiejie Hi, please email @JimmySuen . I am not quite sure.

Ericjiejie commented 4 years ago

Ok, thanks .But I don't know if he will return my email. The reason why I ask is because you say that the output of the network is different from that of GT Heatmap(you talk to @r1ch88), so I think the heatmap on Figure 2 comes from softmax.

lck1201 commented 4 years ago

I guess it's the output of softmax, beacause only the value after softmax represents possibility.

Ericjiejie commented 4 years ago

I agree with you. I have another question. If the output of the network is not like Gaussian heatmap, even very different, why does the author use GT Heatmap for supervision? Will adding this supervision not affect network training?

Ericjiejie commented 4 years ago

@lck1201 Hi, where is the I1 loss(heatmap loss and joint loss) in your code? I want to know the whole process of the integral version(I1), because I have some problems to look for in the code.In detail, I don’t understand the heatmap supervision part and want to know what the output of the network is.

lck1201 commented 4 years ago

check https://github.com/JimmySuen/integral-human-pose/blob/master/pytorch_projects/common_pytorch/common_loss/integral.py

Lixiuxiu2020 commented 4 years ago

I checked it, but there is no heatmap loss, only joint loss.