How to extract coordinates of 3D keypoints

[jin-tmoya]

Hi,I am not good at English, so I apologize if my English is misleading.

I want to extract the coordinates of 3D keypoints(x , y, z), but it doesn't work. As for what we tried, I tried to output the value of prediction.（「videopose-j16-wild-eval_run.py」line 199）

The results are shown here.（We used a "kunkun_clip.mp4".） -------------- prepare video clip spends 0.02 seconds -------------- load keypoint spends 0.01 seconds Loading checkpoint ./checkpoint/ckpt_ep_045.bin -------------- load 3D model spends 0.08 seconds [[[ 5.93354198e-05 1.70869516e-05 3.85639858e-08] [-1.03275985e-01 -2.98896129e-03 3.72205190e-02] [-2.07049251e-01 4.06995893e-01 -1.07592411e-01] ... [-1.62893653e-01 -4.70567942e-01 -5.11920266e-02] [-1.94243610e-01 -2.81453490e-01 1.39230907e-01] [-2.94799685e-01 -1.80989355e-01 1.17846029e-02]]

[[ 6.81496749e-05 1.57090817e-05 4.34617711e-08] [-1.01091936e-01 -2.09802575e-03 4.19775918e-02] [-2.11011797e-01 4.05896515e-01 -9.80924964e-02] ... [-1.53813958e-01 -4.74190831e-01 -3.30972932e-02] [-1.90179735e-01 -3.07276726e-01 1.67530388e-01] [-3.05322289e-01 -2.26756632e-01 6.73069879e-02]]

[[ 7.13446352e-05 1.38120804e-05 4.19068655e-08] [-1.00091867e-01 -7.69968377e-04 4.70813289e-02] [-2.13834763e-01 4.10951346e-01 -9.52427685e-02] ... [-1.51114359e-01 -4.77976769e-01 -2.20416579e-02] [-2.04482198e-01 -3.37212771e-01 1.90199822e-01] [-3.38903546e-01 -2.81002909e-01 1.29762441e-01]]

...

[[-6.72895112e-06 1.62988144e-05 -1.20641790e-08] [-1.03694782e-01 6.92390744e-03 5.06759100e-02] [-1.96537614e-01 4.33968723e-01 -1.27974689e-01] ... [-3.33037019e-01 -3.92906040e-01 -1.15641028e-01] [-5.39204121e-01 -3.08923870e-01 3.72153409e-02] [-7.58739769e-01 -2.83981621e-01 -3.50418091e-02]]

[[ 4.52273525e-08 1.58211951e-05 -1.74119279e-08] [-1.01883560e-01 7.35222874e-03 5.44899777e-02] [-2.18860939e-01 4.36264157e-01 -1.25121012e-01] ... [-3.36887985e-01 -3.93161565e-01 -1.23000994e-01] [-5.36568999e-01 -2.95687914e-01 3.57882604e-02] [-7.49117494e-01 -2.61148334e-01 -4.28803191e-02]]

[[ 1.09871326e-05 1.75449386e-05 -2.58652264e-08] [-1.00538105e-01 9.60641168e-03 5.87101802e-02] [-2.38827333e-01 4.35711741e-01 -1.21683776e-01] ... [-3.37355673e-01 -3.86507511e-01 -1.32569075e-01] [-5.34336150e-01 -2.70994544e-01 2.65108123e-02] [-7.29883015e-01 -2.23541379e-01 -6.84319139e-02]]] -------------- generate reconstruction 3D data spends 0.03 seconds Loading checkpoint ./checkpoint/ckpt_ep_045.bin -------------- load 3D Traj model spends 0.10 seconds [[[-0.0608316 0.06176288 4.3910446 ]]

[[-0.06548614 0.07058202 4.388146 ]]

[[-0.06756089 0.07601852 4.394839 ]]

[[-0.06937461 0.07606728 4.4062023 ]]

[[-0.07935582 0.08046775 4.4164486 ]]

[[-0.08117009 0.08611105 4.454586 ]]

[[-0.08033934 0.08394711 4.507877 ]]

[[-0.08038983 0.08460207 4.5325723 ]]

[[-0.08086307 0.09273802 4.5473223 ]]

[[-0.08537923 0.11209127 4.5626073 ]]

[[-0.0958671 0.14113364 4.5830708 ]]

[[-0.10283907 0.16764934 4.5974092 ]]

[[-0.10631857 0.20077157 4.6156845 ]]

[[-0.11503273 0.22333497 4.6380987 ]]

[[-0.12763707 0.23823354 4.685437 ]]

[[-0.14130223 0.24021342 4.7722874 ]]

[[-0.15757567 0.22182588 4.8492317 ]]

[[-0.17067264 0.19555132 4.9141035 ]]

[[-0.1773699 0.16885322 4.966269 ]]

[[-0.18369074 0.14390603 4.9619675 ]]

[[-0.18909636 0.13704099 4.9318147 ]]

[[-0.19336607 0.13380723 4.8802266 ]]

[[-0.19484845 0.13180962 4.823578 ]]

[[-0.19598952 0.13649382 4.7753596 ]]

[[-0.19962612 0.14402375 4.781396 ]]

[[-0.20210056 0.15149407 4.802383 ]]

[[-0.20000105 0.1506775 4.8272696 ]]

[[-0.20134133 0.14728308 4.8650827 ]]

[[-0.20255381 0.15239106 4.8854446 ]]

[[-0.19994022 0.16628543 4.8886423 ]]

[[-0.19105588 0.17989056 4.8842635 ]]

[[-0.1787905 0.20058975 4.8489604 ]]

[[-0.16615632 0.22787926 4.804517 ]]

[[-0.15367337 0.2493041 4.7636037 ]]

[[-0.14456213 0.25460663 4.743619 ]]

[[-0.13935232 0.24980387 4.7353373 ]]

[[-0.13870691 0.23478699 4.73121 ]]

[[-0.14449558 0.21044308 4.72014 ]]

[[-0.14755204 0.1812517 4.7154474 ]]

[[-0.1413371 0.1611051 4.6990433 ]]

[[-0.1335179 0.1436337 4.6672726 ]]

[[-0.12924318 0.12828264 4.672592 ]]

[[-0.1243439 0.11969504 4.647041 ]]

[[-0.11906209 0.11369721 4.621543 ]]

[[-0.11781553 0.10911711 4.630979 ]]

[[-0.11602639 0.10793802 4.638523 ]]

[[-0.1132953 0.10446329 4.6481276 ]]

[[-0.11169393 0.09817347 4.6719456 ]]

[[-0.11350296 0.09695254 4.6916246 ]]]

I am not sure if this way of putting it out is correct or if it is different in the first place. If you know how to do it right, please let me know.Thank you.

[Garfield-kh]

Hi, thank you for the interest. I think I have saved the prediction 3D pose in this line L247. Is this the one you look for?

[jin-tmoya]

Hi,Thanks for the reply. I have output the results based on your advice.

{'result': array([[[-0.0608316 , 0.06176288, 4.3910446 ], [-0.16416693, 0.05875683, 4.428265 ], [-0.2679402 , 0.4687417 , 4.283452 ], ..., [-0.2237846 , -0.40882215, 4.3398523 ], [-0.25513455, -0.2197077 , 4.5302753 ], [-0.35569063, -0.11924357, 4.402829 ]],

   [[-0.06548614,  0.07058202,  4.388146  ],
    [-0.16664623,  0.06846829,  4.4301233 ],
    [-0.2765661 ,  0.47646284,  4.2900534 ],
    ...,
    [-0.21936825, -0.4036245 ,  4.3550487 ],
    [-0.25573403, -0.2367104 ,  4.5556765 ],
    [-0.37087658, -0.1561903 ,  4.455453  ]],

   [[-0.06756089,  0.07601852,  4.394839  ],
    [-0.1677241 ,  0.07523474,  4.4419203 ],
    [-0.281467  ,  0.48695606,  4.299596  ],
    ...,
    [-0.2187466 , -0.40197206,  4.372797  ],
    [-0.27211443, -0.26120806,  4.5850387 ],
    [-0.40653577, -0.2049982 ,  4.524601  ]],

Sorry for the additional question, but I am concerned that the z-coordinate is "4" and hardly changes at all. If you try to plot this as it is, it seems like the z-axis would be uniform, but the actual output 3D figure shows a proper 3D figure. Why is this? Is there some special processing that has been done to the coordinate values?Thank you.

[Garfield-kh]

Hi, the z-coordinate here is under camera-coordinate (Y-up, Z-front), means "4" is 4 meter away from camera.

[jin-tmoya]

Hi,this is the first time I heard about (Y-up, Z-front),It seems to be different from the z-coordinates I had in mind. I would say, I would like to produce an accurate arm angle based on the output coordinates.I would expect that your system, which can reproduce spatially with a monocular camera, would be able to accurately produce the angle of the arm when it is bent facing forward. Is this possible? Thank you.

[Garfield-kh]

I didn't measure the accuracy of arm angle specifically. I think you can try take the test set with the case you are interested, and measure them with ground truth specifically on arm angle. Meanwhile, the depth indeed is the most hard part in pose estimation due to the depth ambiguity.

[jin-tmoya]

I see. I have seen many different pose estimatation and find it difficult to measure depth. Nevertheless, I feel that the 3Dization by your system is accurately producing depth. Is there any way to get the depth of each joint point, even if it is just an estimate?Thank you.

[Garfield-kh]

The predicted pose X is tx16x3. Depth should be the z value is X[:,:,2].

[jin-tmoya]

Thank you for telling us. Pardon my ignorance, I would like to ask two questions.

1.What do "tx16x3" stand for respectively? My perception is that"16" is the number of key points."3" is the coordinate or color?

2.Where in the code could X[:,:,2] be output from? I think it is L180

[bingbing-li]

t frames 16 key points 3 – <x,y,z> coordinates

From: jin-tmoya @.> Sent: 2022年10月25日 8:22 To: Garfield-kh/PoseTriplet @.> Cc: Subscribed @.***> Subject: Re: [Garfield-kh/PoseTriplet] How to extract coordinates of 3D keypoints (Issue #24)

Thank you for telling us. Pardon my ignorance, I would like to ask two questions.

1.What do "tx16x3" stand for respectively? My perception is that"16" is the number of key points."3" is the coordinate or color?

2.Where in the code could X[:,:,2] be output from? I think it is L180https://github.com/Garfield-kh/PoseTriplet/blob/eb93132f99161bd776dafbcb713e9fb43e501c36/estimator_inference/common/visualization.py#L180

— Reply to this email directly, view it on GitHubhttps://github.com/Garfield-kh/PoseTriplet/issues/24#issuecomment-1289809682, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AD67KDJL32B6ESI4YMVZDBDWE4R2LANCNFSM6AAAAAARIZQCDI. You are receiving this because you are subscribed to this thread.Message ID: @.***>

[jin-tmoya]

Thanks for telling us about tx16x3. Sorry, my English is not good enough, so let me reconfirm. Would it be correct if I email Mr. bingbing-li regarding future correspondence?

[bingbing-li]

Hi, No worries about English as long as there is no misunderstandings.

You may still reply in the thread. We are specialized in different direction, so anyone of us may reply your at our best.

From: jin-tmoya @.> Sent: 2022年10月27日 7:19 To: Garfield-kh/PoseTriplet @.> Cc: #LI BINGBING# @.>; Comment @.> Subject: Re: [Garfield-kh/PoseTriplet] How to extract coordinates of 3D keypoints (Issue #24)

Thanks for telling us about tx16x3. Sorry, my English is not good enough, so let me reconfirm. Would it be correct if I email Mr. bingbing-li regarding future correspondence?

— Reply to this email directly, view it on GitHubhttps://github.com/Garfield-kh/PoseTriplet/issues/24#issuecomment-1292765627, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AD67KDLRXASL5A5MRWIR6B3WFG35LANCNFSM6AAAAAARIZQCDI. You are receiving this because you commented.Message ID: @.***>

[jin-tmoya]

I felt bad to repeat my contact in the thread, so I have sent you an email. Thank you.

[jin-tmoya]

I'm sorry, did you not receive my email? If you did not receive it, I would like to post it again in this thread.