TRC file question on OpenSim

[bin851007]

Hello David,

I'm Jack.

I am trying to test your pose2sim program.

I'm using 5 camera Calibration but the results are not great, both int and ext are over 0.5px, I choose to ignore it :)

Then using the triangulation equation for OpenPose json of Body_25, I get an error of less than 15px, so,I filter trc file.

When using OpenSim, I have no problem of scaling, but using IK，I get upside down and jittery or distorted motion. I think the reverse can be solved by conversion, but there is no other solution.

I have sent you the Empty_project file to you(contact@david-pagnon.com), which contains my data ,basic information, andtrc file. If you see it, I will be very happy, because it will be a big help for my research.

my emailaddress (bin851007@gmail.com)

[davidpagnon]

Hi there, sorry I'm kind of in a rush right now but I'll check it by the end of this week or the following!

[davidpagnon]

So I found a bit of time to have a look at your folder :)

• First of all, the pictures you took for intrinsic calibration are perfect: they take enough space on the camera frame and are shown from different angles. However, I can see that cameras 1, 3, and 5 have a rather large distortion, which is not well taken into account in triangulation. This is on the to-do list, not really hard to fix, but I haven't found time to do it yet. This may be the reason why your intrinsic parameter's error is sometimes above 0.5 px. Aside from camera 3, it is really close so I would consider intrinsic calibration to be successful. Do you, by any chance, have a "linear" mode like there is in GoPro cameras? If not, I may try to make the change in the next few weeks.

• Concerning your extrinsic calibration, the calibration board is tiny compared to the volume you want to calibrate, and points are really hard to click accurately. I would use either a much bigger board for calibration, or just use coordinates on the scene. I can see that you have tiles and pieces of furniture in your room. You can give each corner a number, measure their coordinates, and put them in object_coords_3d, and label them with the calibration tool. You will likely get better results.

• The reason I could see for having an upside-down IK would be that you labeled the corners in the wrong order during extrinsic calibration. Have a look at how they are labeled in the intrinsic calibration and go in the same order.

Keep me updated!

[davidpagnon]

Hi @bin851007, did you have a chance to try it again?

[bin851007]

Sorry David, I was busy writing my thesis,so I forget to reply you.

I tried it again and got a different result,it's better than the first one.

But there are still some littile problems.

I will respond on github

Maybe I transfor dates for you first.

David PAGNON @.***> 於 2023年12月27日 週三 上午12:36寫道：

Hi @bin851007 https://github.com/bin851007, did you have a chance to try it again?

— Reply to this email directly, view it on GitHub https://github.com/perfanalytics/pose2sim/issues/55#issuecomment-1869652625, or unsubscribe https://github.com/notifications/unsubscribe-auth/AY3JZCWZPOM45QGMKDATMVDYLL4HVAVCNFSM6AAAAABAQME72GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRZGY2TENRSGU . You are receiving this because you were mentioned.Message ID: @.***>

[davidpagnon]

Okay, tell me when you need it!

[bin851007]

I don’t have any GoPro, or other cameras, so I can only use the same camera to calibrate. I followed your suggestion and used scenes for extrinsic calibration. Although, it wasn’t perfect, it was better than the first time. For intrinsic calibration, I code a simple camera calibration with Python to get distortions, and copied them to Calib_toml, then,the errors reduced 1~2px, but it still couldn't get less than 0.5px.Also, I used the Matlab Calibration app, but there were no distortions p1 p2 , so I gave up. Thus, there is still no solution for intrinsic calibration.

Regarding IK, I successfully generated the correct pose, which makes me happy, but there are still some little problems, such as the head and left toe are in the wrong site, or the body looks it's shaking. Is there any solution?

I also sent the data for you :)

[davidpagnon]

Okay, so let's talk about each step one after the other:

• Are your 2D pose estimation results correct? OpenPose does not always work fine when the person is seen from above, below, or with the person lying horizontally.

• Concerning calibration, with such large distortions, you won't be able to get perfect results. However, 4.5 cm error is not great but not too bad. From the "detailed description" in the OpenCV documentation, you can just use the distortion parameters obtained with your method and set the others to zero.

  In the functions below the coefficients are passed or returned as (k1,k2,p1,p2[,k3[,k4,k5,k6[,s1,s2,s3,s4[,τx,τy]]]]) vector. That is, if the vector contains four elements, it means that k3=0 .

  However, I doubt this is the problem. With my method, the distortion parameters are calculated at the calibration phase, but are not correctly taken into account in the IK phase when the camera is a fisheye:

  On the other hand, your 3rd camera has issues, and right now, triangulation is only going to use 4 of them. It could be caused by you not clicking the extrinsic points in the right order for example.

• IK seems to be mostly okay, despite it using only 4 cameras. Did you try to filter your results?

Overall, I would tend to think your OpenPose results might actually be the main issue. Maybe you could try to put your cameras more at chest level?

[davidpagnon]

Hi @bin851007, I used your data as an example to better handle camera distortions (beware that currently, it is still better to leave that undistort_points parameter to false), and I got better calibration results with my tool, even when calibrating intrinsic parameters with the board:

--> Residual (RMS) calibration errors for each camera are respectively [1.711, 1.21, 1.867, 1.217, 1.775] px, 
which corresponds to [7.088, 3.876, 6.105, 4.654, 7.166] mm.

If you want the person to not be upside down, you need to click the checkerboard points while keeping in mind the "right-hand rule": the first row of corners will represent your X axis, the columns will be the Y axis, and they need to be chosen so that the Z axis points up.

The motion looks rather good despite I did not take distortions into account, and despite some points on the feet being occluded at the deepest point of your squatting motion.

I'm still hoping I will find time to better handle distortions, but right now I need to work on other stuff. Just in case, make sure you upgrade to the last version by entering pip install pose2Sim --upgrade.

[davidpagnon]

Handled by email