search

DenisTome / Lifting-from-the-Deep-release

Implementation of "Lifting from the Deep: Convolutional 3D Pose Estimation from a Single Image"
https://denistome.github.io/papers/lifting-from-the-deep
GNU General Public License v3.0
450 stars 133 forks source link

Skewed y coordinates in 3d pose #20

Closed rludlow closed 6 years ago

rludlow commented 6 years ago

Hi Denis - Thank you for sharing this project.

I've run into an issue in which the model outputs highly skewed y coordinates for certain images. For example, the model outputs a reasonable re-construction of the first frame (img) in this sequence:

lift0_display 
pose_3d = [[[ 165.8230328    67.03754957 -205.89986905  -32.38983883  302.10612467
     91.22563632  280.07067535   64.23923688 -121.04723249 -166.27962564
   -226.57145921   80.90181175  263.25173031  314.5440196  -233.47394329
   -260.15381899 -319.60138574]
  [ 209.65279238  314.71824629   82.10550074  165.22295977  104.58738359
   -168.25258589   13.24661237   87.6175417  -111.29930409 -200.36746041
   -205.78091077 -181.83588106 -160.04598113 -278.14880424   28.59255825
    181.96800836   99.68732301]
  [ -29.95290344  -30.35298069   -5.77654454 -337.00434106    6.86051195
   -103.72445841 -435.77584961  155.03801527  311.75958399  306.73182102
    533.42573399  223.34929849  -75.80560911 -363.73045755  260.07321576
     10.41625411 -163.05433098]]]

However, in other frames, the x and z coordinates remain consistent with similar frames, but the y coordinates are dramatically skewed, such as here (img):

lift32_display 
pose_3d = [[[ 1.02271246e+02  9.62935992e+01 -2.34439449e+02 -1.05166807e+02
    2.98286838e+02  1.64104641e+02  3.41578049e+02 -1.27844847e+01
   -1.25346562e+02 -1.31937004e+02 -3.06537751e+02  2.64495353e+00
    2.90037744e+02  4.65259697e+02 -2.01997730e+02 -3.24498281e+02
   -3.60220340e+02]
  [-7.38312954e+04 -7.38708949e+04 -7.38310055e+04 -7.38060559e+04
   -7.37916961e+04 -7.37781771e+04 -7.38124515e+04 -7.36989474e+04
   -7.35605069e+04 -7.36014401e+04 -7.35446308e+04 -7.35147100e+04
   -7.34832985e+04 -7.35327440e+04 -7.36254187e+04 -7.39279633e+04
   -7.40327591e+04]
  [-2.10067792e+02 -2.62270589e+02 -3.37599039e+02 -7.12890082e+02
   -2.10924765e+02 -4.33412911e+02 -8.77269711e+02 -1.90837834e+01
    1.60809907e+02  3.17632726e+02  3.83645266e+02  1.99421824e+02
    3.73765312e+02  5.77611265e+02  1.04109928e+02  3.57356140e+02
    6.42948964e+02]]]

The y coordinates are showing up skewed by 2 orders of magnitude and some other factor I can't determine. This happens for around 20% of the images that I try. Do you have any idea what the problem is and how I could address it?

DenisTome commented 6 years ago

Hi, the reason should be related to the uncertainty connected to the 2D joints shown in the image. Even though the joints are all shown, some of them might have a low uncertainty when predicted, and the 3D module does not consider them reliable enough to be taken into account. Since this approach has to work with images in the wild, it's quite difficult to choose the right uncertainty threshold appropriate for a specific set of images; the default one at the time looked like a sensibe choice. However, you can play yourself by changing the threshold located at lifting/utils/config.py, in particular VISIBLE_PART and MIN_NUM_JOINTS

rludlow commented 6 years ago

Thanks Denis. I tried adjusting VISIBLE_PART and MIN_NUM_JOINTS and didn't see a difference.

Edit: That said, I now see how it is indeed an issue of joint/pose uncertainty.

DenisTome commented 6 years ago

Ok. Regardless, about the constant problem you were taking about, if you use plot_pose(Prob3dPose.centre_all(single_3D)) then this should fix the problem