Closed usamahjundia closed 4 years ago
There are too many questions and actually, answers of most of the questions are in my paper. Could you reduce the number of questions after reading the paper?
No, i quite understand the gist of some questions actually (which is pretty much explained clearly in the paper, thanks for that), but not the technical aspect of doing so. Here are the reduced actual questions:
Thank you. I apologize for asking too many questions as i asked the questions while reading the paper and i really need a lot of iteration in reading the paper and as you observe, i edited the questions several times as i got answers in an iteration of reading.
What i meant by the 3rd point is to represent the depth/distance (the corrected K value) in pixel dimension. But it seems like i found some insight in the code. I'm looking at this line here in MSCOCO.py:
pred_2d_kpt = np.take(pred_2d_kpt, self.eval_joint, axis=0)
pred_2d_kpt[:,0] = pred_2d_kpt[:,0] / cfg.output_shape[1] * bbox[2] + bbox[0]
pred_2d_kpt[:,1] = pred_2d_kpt[:,1] / cfg.output_shape[0] * bbox[3] + bbox[1]
pred_2d_kpt[:,2] = (pred_2d_kpt[:,2] / cfg.depth_dim * 2 - 1) * (cfg.bbox_3d_shape[0]/2) + gt_3d_root[2]
Seems like PoseNet outputs all x,y,z in pixel unit, and this is the conversion into length unit.
So is this statement true:
In 3D pixel coordinates, humans are surrounded by a 3D bounding box of size 256x256x256 (or 64?) and in real world coordinates of size 2x2x2 meters, and the processing on the third dimension of posenet outputs is the mapping from this 256^3 in pixel space to 2^3 in length unit (meter) space?
(Yes, i am aware this isnt the repo for posenet, but starting a new issue in the other repo just to get a related question answered does not seem effective)
First of all, depth cannot be converted to pixel. Depth is defined in z-axis, and pixel is defined in x- and y-axis. PoseNet outputs x,y in image coordinate space and z in discretized camera-centered space. Regarding your question, there is no 3D pixel coordinate. Do you mean voxel? If that is the case, the answer is close to yes. I cannot understand 'processing on the third dimension'
by processing on the third dimension i meant this line
pred_2d_kpt[:,2] = (pred_2d_kpt[:,2] / cfg.depth_dim * 2 - 1) * (cfg.bbox_3d_shape[0]/2) + gt_3d_root[2]
and okay, i obviously failed trying to frame it in academic terms so let's reframe: let's say i want to visualize or export the 3D coordinates of a pose, for example, so that i can use it to animate a 3D model of a character. How do i make sure the x,y, and z coordinates are consistent, as in, movements along the x axis and y axis (which is in posenet outputs, in terms of image space which i have been failing to find the right term for) is in the same scale / unit as the z axis?
L168 of https://github.com/mks0601/3DMPPE_POSENET_RELEASE/blob/master/data/MSCOCO/MSCOCO.py
pixel2cam
converts (x_img, y_img, z_cam) to (x_cam, y_cam, z_cam)
Hello! Thanks in advance for this implementation of a cool paper! Currently i am trying to combine both rootnet and posenet into a singular pipeline for inference into 3D. What i want to ask is, what is the combination strategy for both? What do we do with both information? How do we extract crop from posenet to input into rootnet? What if there is a problem with occlussion, etc? Given the paper uses Mask-RCNN to first extract individual humans, do we mask out the unnecessary regions from a singular crop?
Then what do we do with posenet and rootnet prediction? Do we just merely add them together? Because judging from the results in the paper, the x,y,z produced by posenet and rootnet is in pixels, right?
Also, rootnet requires a pre-computed K value, how did you get this value?
Thanks!