search

princeton-vl / pytorch_stacked_hourglass

Pytorch implementation of the ECCV 2016 paper "Stacked Hourglass Networks for Human Pose Estimation"
BSD 3-Clause "New" or "Revised" License
465 stars 94 forks source link

reversed (left&right) shoulder ground truth #9

Closed Zepyhrus closed 4 years ago

Zepyhrus commented 4 years ago

Hi all, thanks for your excellent work.

When I was trying to browse the validation data, I realized some ground truth are marked reversely, typically shoulders (point 12&13), will this deteriorate model performance?

if this does have effects on the performance, I will consider revising it.

Sample image comes with 024240973.jpg and see my attachment bellow.

BR

__screenshot_24 02 2020

Zepyhrus commented 4 years ago

BTW, not much related to this topic but may help with some successors, I have moved from scipy.misc to opencv, and this DOES deteriorate model performance slightly, around less than 1%.

crockwell commented 4 years ago

Have you checked MPII annotations directly? I believe our files should be consistent with theirs. If this is the case, it is possible some are mislabeled -- human annotators can occasionally make mistakes. Thanks!

It is possible implementation of operations for pre/post-processing are slightly different between scipy and opencv, so probably best to be consistent with what we used for training when testing (I'm guessing this is why you see this dropoff).

Zepyhrus commented 4 years ago

Have you checked MPII annotations directly? I believe our files should be consistent with theirs. If this is the case, it is possible some are mislabeled -- human annotators can occasionally make mistakes. Thanks!

It is possible implementation of operations for pre/post-processing are slightly different between scipy and opencv, so probably best to be consistent with what we used for training when testing (I'm guessing this is why you see this dropoff).

Sorry, I didn't check the MPII annotations directly, only from the .h5 file provided in this project. From my investigation, there are around 7~10% people are mislabeled (only checked the train.h5 file), will this lead to a miscalculated loss from the output? Cause this is pretty much.

crockwell commented 4 years ago

That is unfortunate. Though, datasets are human annotated and thus can contain mistakes, so it is certainly possible there are some "flips" in MPII. If you think this is specific to our h5 file I would be curious to hear! It might be worth checking MPII repo to see if this is a known issue, I have noticed some mistakes while running our code, though 7-10% might sound a high to me.